Introduction to Redpanda Connect

Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. It implements transaction-based resiliency with back pressure, so when connecting to at-least-once sources and sinks it’s able to guarantee at-least-once delivery without needing to persist messages during transit.

It’s simple to deploy, comes with a wide range of connectors, and is totally data agnostic, making it easy to drop into your existing infrastructure. Redpanda Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative.

Components



Name Connector Type Support Level Enterprise Licensed Available in Cloud

amqp_0_9

Input, Output

Certified

No

Yes

amqp_1

Input, Output

Community

No

No

archive

Processor

Certified

No

Yes

avro

Processor, Scanner

Community

No

Yes

awk

Processor

Community

No

No

aws_bedrock_chat

Processor

Certified

Yes

Yes

aws_bedrock_embeddings

Processor

Certified

Yes

Yes

aws_cloudwatch

Metric

Community

No

No

aws_dynamodb

Cache, Output

Community

No

Yes

aws_dynamodb_partiql

Processor

Certified

No

Yes

aws_kinesis

Input, Output

Certified

No

Yes

aws_kinesis_firehose

Output

Certified

No

Yes

aws_lambda

Processor

Certified

No

Yes

aws_s3

Cache, Input, Output

Certified

No

Yes

aws_sns

Output

Community

No

Yes

aws_sqs

Input, Output

Certified

No

Yes

azure_blob_storage

Input, Output

Certified

No

Yes

azure_cosmosdb

Input, Output, Processor

Certified

No

Yes

azure_data_lake_gen2

Output

Certified

No

Yes

azure_queue_storage

Input, Output

Certified

No

Yes

azure_table_storage

Input, Output

Certified

No

Yes

batched

Input

Certified

No

Yes

beanstalkd

Input, Output

Community

No

No

bloblang

Processor

Certified

No

Yes

bounds_check

Processor

Certified

No

Yes

branch

Processor

Certified

No

Yes

broker

Input, Output

Certified

No

Yes

cache

Output, Processor

Certified

No

Yes

cached

Processor

Certified

No

Yes

cassandra

Input, Output

Community

No

No

catch

Processor

Certified

No

Yes

chunker

Scanner

Certified

No

Yes

cockroachdb_changefeed

Input

Community

No

No

cohere_chat

Processor

Certified

Yes

Yes

cohere_embeddings

Processor

Certified

Yes

Yes

command

Processor

Certified

No

No

compress

Processor

Certified

No

Yes

couchbase

Cache, Output, Processor

Community

No

No

csv

Input, Scanner

Certified

No

Yes

cypher

Output

Community

No

No

decompress

Processor, Scanner

Certified

No

Yes

dedupe

Processor

Certified

No

Yes

discord

Input, Output

Community

No

No

drop

Output

Certified

No

Yes

drop_on

Output

Certified

No

Yes

dynamic

Input, Output

Community

No

No

elasticsearch

Output

Community

No

No

fallback

Output

Certified

No

Yes

file

Cache, Input, Output

Certified

No

No

for_each

Processor

Certified

No

Yes

gcp_bigquery

Output

Certified

No

Yes

gcp_bigquery_select

Input, Processor

Certified

No

Yes

gcp_cloud_storage

Cache, Input, Output

Certified

No

Yes

gcp_cloudtrace

Tracer

Certified

No

Yes

gcp_pubsub

Input, Output

Certified

No

Yes

gcp_vertex_ai_chat

Processor

Certified

Yes

Yes

gcp_vertex_ai_embeddings

Processor

Certified

Yes

Yes

generate

Input

Certified

No

Yes

grok

Processor

Community

No

No

group_by

Processor

Certified

No

Yes

group_by_value

Processor

Certified

No

Yes

hdfs

Input, Output

Community

No

No

http

Processor

Certified

No

Yes

http_client

Input, Output

Certified

No

Yes

http_server

Input, Output

Certified

No

No

influxdb

Metric

Community

No

No

inproc

Input, Output

Certified

No

Yes

insert_part

Processor

Certified

No

Yes

jaeger

Tracer

Community

No

No

javascript

Processor

Certified

No

No

jmespath

Processor

Certified

No

Yes

jq

Processor

Certified

No

Yes

json_api

Metric

Certified

No

No

json_documents

Scanner

Certified

No

Yes

json_schema

Processor

Certified

No

Yes

kafka

Input, Output

Certified

No

Yes

kafka_franz

Input, Output

Certified

No

Yes

lines

Scanner

Certified

No

Yes

local

Rate_limit

Certified

No

Yes

log

Processor

Certified

No

Yes

logger

Metric

Certified

No

No

lru

Cache

Community

No

Yes

mapping

Processor

Certified

No

Yes

memcached

Cache

Community

No

Yes

memory

Buffer, Cache

Certified

No

Yes

metric

Processor

Certified

No

Yes

mongodb

Cache, Input, Output, Processor

Community

No

No

mqtt

Input, Output

Certified

No

Yes

msgpack

Processor

Community

No

No

multilevel

Cache

Certified

No

Yes

mutation

Processor

Certified

No

Yes

nanomsg

Input, Output

Community

No

No

nats

Input, Output

Certified

No

Yes

nats_jetstream

Input, Output

Certified

No

Yes

nats_kv

Cache, Input, Output, Processor

Certified

No

Yes

nats_request_reply

Processor

Certified

No

Yes

nats_stream

Input, Output

Community

No

No

none

Buffer, Metric, Tracer

Certified

No

Yes

noop

Cache, Processor

Certified

No

Yes

nsq

Input, Output

Community

No

No

ockam_kafka

Community

No

No

ollama_chat

Processor

Certified

Yes

No

ollama_embeddings

Processor

Certified

Yes

No

open_telemetry_collector

Tracer

Community

No

No

openai_chat_completion

Processor

Certified

Yes

Yes

openai_embeddings

Processor

Certified

Yes

Yes

openai_image_generation

Processor

Certified

Yes

Yes

openai_speech

Processor

Certified

Yes

Yes

openai_transcription

Processor

Certified

Yes

Yes

openai_translation

Processor

Certified

Yes

Yes

opensearch

Output

Certified

No

Yes

parallel

Processor

Certified

No

Yes

parquet

Input, Processor

Certified: input

Community: processor

No

No

parquet_decode

Processor

Certified

No

Yes

parquet_encode

Processor

Certified

No

Yes

parse_log

Processor

Community

No

Yes

pg_stream

Community

No

No

pinecone

Output

Certified

No

Yes

processors

Processor

Certified

No

Yes

prometheus

Metric

Certified

No

Yes

protobuf

Processor

Certified

No

Yes

pulsar

Input, Output

Community

No

No

pusher

Output

Community

No

No

qdrant

Output

Certified

No

Yes

questdb

Output

Certified

No

Yes

rate_limit

Processor

Certified

No

Yes

re_match

Scanner

Certified

No

Yes

read_until

Input

Certified

No

Yes

redis

Cache, Processor, Rate_limit

Certified

No

Yes

redis_hash

Output

Certified

No

Yes

redis_list

Input, Output

Certified

No

Yes

redis_pubsub

Input, Output

Certified

No

Yes

redis_scan

Input

Certified

No

Yes

redis_script

Processor

Certified

No

Yes

redis_streams

Input, Output

Certified

No

Yes

redpanda

Input, Output

Certified

No

Yes

redpanda_common

Input, Output

Certified

Yes

Yes

redpanda_data_transform

Processor

Certified

No

No

redpanda_migrator

Input, Output

Certified

Yes

Yes

redpanda_migrator_bundle

Input, Output

Certified

Yes

Yes

redpanda_migrator_offsets

Output

Certified

Yes

Yes

reject

Output

Certified

No

Yes

reject_errored

Output

Certified

No

Yes

resource

Input, Output, Processor

Certified

No

Yes

retry

Output, Processor

Certified

No

Yes

ristretto

Cache

Community

No

Yes

schema_registry

Input, Output

Certified

Yes

Yes

schema_registry_decode

Processor

Certified

No

Yes

schema_registry_encode

Processor

Certified

No

Yes

select_parts

Processor

Certified

No

Yes

sentry_capture

Processor

Community

No

No

sequence

Input

Certified

No

Yes

sftp

Input, Output

Certified

No

Yes

skip_bom

Scanner

Certified

No

Yes

sleep

Processor

Certified

No

Yes

snowflake_put

Output

Certified

Yes

Yes

snowflake_streaming

Output

Certified

Yes

Yes

socket

Input, Output

Certified

No

No

socket_server

Input

Certified

No

No

spicedb_watch

Input

Community

No

Yes

split

Processor

Certified

No

Yes

splunk

Input

Certified

Yes

Yes

splunk_hec

Output

Certified

Yes

Yes

sql

Cache, Output, Processor

Certified: cache

Community: output, processor

No

No

sql_driver_clickhouse

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

No

sql_driver_gocosmos

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

No

sql_driver_mssql

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

No

sql_driver_mysql

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

Yes

sql_driver_oracle

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

Yes

sql_driver_postgres

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

Yes

sql_driver_snowflake

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

No

sql_driver_sqlite

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

Yes

sql_driver_trino

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

No

sql_insert

Output, Processor

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

Yes

sql_raw

Input, Output, Processor

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

Yes

sql_select

Input, Processor

Certified: MYSQL, Oracle, PostgreSQL, SQLite
Community: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino

No

Yes

sqlite

Buffer

Community

No

No

statsd

Metric

Certified

No

No

stdin

Input

Certified

No

No

stdout

Output

Certified

No

No

subprocess

Input, Output, Processor

Community

No

No

switch

Output, Processor, Scanner

Certified

No

Yes

sync_response

Output, Processor

Certified

No

Yes

system_window

Buffer

Certified

No

Yes

tar

Scanner

Certified

No

Yes

timeplus

Input, Output

Community

No

Yes

to_the_end

Scanner

Certified

No

Yes

try

Processor

Certified

No

Yes

ttlru

Cache

Community

No

Yes

twitter_search

Input

Community

No

No

unarchive

Processor

Certified

No

Yes

wasm

Processor

Community

No

No

websocket

Input, Output

Certified

No

No

while

Processor

Certified

No

Yes

workflow

Processor

Certified

No

Yes

xml

Processor

Community

No

Yes

zmq4

Input, Output

Community

No

No