Connect

Components Catalog

Use the following table to search for available inputs, outputs, and processors.

For a description of the support levels, see Support levels.

memory

Stores consumed messages in memory and acknowledges them at the input level.

Type
Support
Certified
Availability

none

Do not buffer messages. This is the default and most resilient configuration.

Support
Certified
Availability

sqlite

Stores messages in an SQLite database and acknowledges them at the input level.

Type
Support
Community

system_window

Chops a stream of messages into tumbling or sliding windows of fixed temporal size, following the system clock.

Type
Support
Certified
Availability

aws_dynamodb

Stores key/value pairs as a single document in a DynamoDB table.

Type
Support
Community
Availability
AWS DynamoDB Amazon DynamoDB DynamoDB

aws_s3

Stores each item in an S3 bucket as a file, where an item ID is the path of the item within the bucket.

Support
Certified
Availability
AWS S3 Amazon S3 S3 Simple Storage Service

couchbase

Use a Couchbase instance as a cache.

Support
Community

file

Stores each item in a directory as a file, where an item ID is the path relative to the configured directory.

Support
Certified

gcp_cloud_storage

Use a Google Cloud Storage bucket as a cache.

Support
Certified
Availability
GCP Cloud Storage Google Cloud Storage GCS

lru

Stores key/value pairs in a lru in-memory cache. This cache is therefore reset every time the service restarts.

Type
Support
Community
Availability

memcached

Connects to a cluster of memcached services, a prefix can be specified to allow multiple cache types to share a memcached cluster under different...

Type
Support
Community
Availability

mongodb

Use a MongoDB instance as a cache.

Support
Certified
Availability
Mongo

multilevel

Combines multiple caches as levels, performing read-through and write-through operations across them.

Type
Support
Certified
Availability

nats_kv

Caches key/value pairs in a NATS key-value bucket.

Support
Certified
Availability
NATS KV

noop

Noop is a cache that stores nothing, all gets returns not found. Why? Sometimes doing nothing is the braver option.

Support
Certified
Availability

redis

Use a Redis instance as a cache. The expiration can be set to zero or an empty string in order to set no expiration.

Support
Certified
Availability

redpanda

A Kafka cache implemented using the Franz Kafka client library.

Support
Certified
Availability

ristretto

Stores key/value pairs in a map held in the memory-bound Ristretto cache.

Type
Support
Community
Availability

sql

Uses an SQL database table as a destination for storing cache key/value items.

Support
Certified Community
Availability

ttlru

Stores key/value pairs in a ttlru in-memory cache. This cache is therefore reset every time the service restarts.

Type
Support
Community
Availability

amqp_0_9

Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ.

Type
Support
Certified
Availability
RabbitMQ AMQP

amqp_1

Reads messages from an AMQP (1.0) server.

Type
Support
Community
RabbitMQ AMQP Apache Qpid

aws_cloudwatch_logs

Consumes log events from AWS CloudWatch Logs.

Type
Support
Community
Availability
AWS CloudWatch Logs Amazon CloudWatch Logs

aws_dynamodb_cdc

Reads change data capture (CDC) events from DynamoDB Streams.

Type
Support
Certified
Availability
Cloud Enterprise
Amazon DynamoDB CDC

aws_kinesis

Receive messages from one or more Kinesis streams.

Type
Support
Certified
Availability
AWS Kinesis Amazon Kinesis Kinesis

aws_sqs

Consume messages from an AWS SQS URL.

Type
Support
Certified
Availability
AWS SQS Amazon SQS SQS Simple Queue Service

azure_blob_storage

Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix.

Type
Support
Certified
Availability
Azure Blob Storage Microsoft Azure Storage

azure_cosmosdb

Executes a SQL query against Azure CosmosDB and creates a batch of messages from each page of items.

Support
Certified
Availability
Microsoft Azure Azure

azure_queue_storage

Dequeue objects from an Azure Storage Queue.

Type
Support
Certified
Availability
Azure Queue Storage Microsoft Azure Queue

azure_table_storage

Queries an Azure Storage Account Table, optionally with multiple filters.

Type
Support
Certified
Availability
Azure Table Storage Microsoft Azure Table

batched

Consumes data from a child input and applies a batching policy to the stream.

Type
Support
Certified
Availability

beanstalkd

Reads messages from a Beanstalkd queue.

Type
Support
Community

broker

Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel.

Type
Support
Certified
Availability

cassandra

Executes a find query and creates a message for each row received.

Type
Support
Community
Apache Cassandra

cockroachdb_changefeed

Listens to a CockroachDB Core Changefeed and creates a message for each row received.

Type
Support
Community

csv

Reads one or more CSV files as structured records following the format described in RFC 4180.

Type
Support
Certified
Availability
Comma-Separated Values

discord

Consumes messages posted in a Discord channel.

Type
Support
Community

dynamic

A special broker type where the inputs are identified by unique labels and can be created, changed and removed during runtime via a REST HTTP...

Type
Support
Community

gateway

Receive messages delivered over HTTP.

Type
Support
Certified
Availability
Cloud Enterprise

gcp_bigquery_select

Executes a SELECT query against BigQuery and creates a message for each row received.

Support
Certified
Availability
GCP BigQuery Google Cloud GCP

gcp_pubsub

Consumes messages from a GCP Cloud Pub/Sub subscription.

Type
Support
Certified
Availability
GCP PubSub Google Cloud Pub/Sub GCP Pub/Sub Google Pub/Sub

gcp_spanner_cdc

Consumes change records from a Google Cloud Spanner change stream.

Type
Support
Certified
Availability
Cloud Enterprise
Google Cloud GCP

generate

Generates messages at a given interval using a Bloblang mapping executed without a context.

Type
Support
Certified
Availability

git

Clones a Git repository, reads its contents, then polls for new commits at a configurable interval.

Type
Support
Certified
Availability

hdfs

Reads files from a HDFS directory, where each discrete file will be consumed as a single message payload.

Type
Support
Community

http_client

Connects to a server and continuously performs requests for a single message.

Type
Support
Certified
Availability
HTTP REST API REST

http_server

Receive messages POSTed over HTTP(S). HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified.

Type
Support
Certified
Availability
HTTP REST API REST Gateway

inproc

Directly connects to an output within the same process by a chosen ID, to link isolated streams when running in streams mode.

Type
Support
Certified
Availability

jira

Streams Jira issues, comments, or changelog entries via JQL with incremental polling.

Support
Certified
Availability
Atlassian Jira

kafka

Connects to Kafka brokers and consumes one or more topics.

Type
Support
Certified
Availability
Apache Kafka

kafka_franz

A Kafka input using the Franz Kafka client library.

Type
Support
Certified
Availability
Apache Kafka Kafka

microsoft_sql_server_cdc

Enables Change Data Capture by consuming from Microsoft SQL Server's change tables.

Type
Support
Certified
Availability
Cloud Enterprise

mongodb_cdc

Streams changes from a MongoDB replica set.

Type
Support
Certified
Availability
Cloud Enterprise
MongoDB CDC

mqtt

Subscribe to topics on MQTT brokers.

Type
Support
Certified
Availability

mysql_cdc

Streams data changes from a MySQL or MariaDB database, using the binary log to capture data updates.

Type
Support
Certified
Availability
Cloud Enterprise

nanomsg

Consumes messages via Nanomsg sockets (scalability protocols).

Type
Support
Community

nats

Subscribes to a NATS subject.

Type
Support
Certified
Availability
NATS.io

nats_jetstream

Reads messages from NATS JetStream subjects.

Type
Support
Certified
Availability
NATS JetStream NATS

nats_stream

Subscribes to a NATS Stream subject. Joining a queue is optional and allows multiple clients of a subject to consume using queue semantics.

Type
Support
Community
NATS Stream NATS Streaming NATS

nsq

Subscribe to an NSQ instance topic and channel.

Type
Support
Community

ockam_kafka

Uses Ockam to decrypt and read end-to-end encrypted messages from Kafka topics.

Type
Support
Community

oracledb_cdc

Enables Change Data Capture by consuming from OracleDB.

Type
Support
Certified
Availability
Cloud Enterprise
Oracle CDC OracleDB CDC Oracle Database CDC

otlp_grpc

Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol.

Type
Support
Certified
Availability
Cloud Enterprise
OpenTelemetry OTLP OTel gRPC

otlp_http

Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol.

Type
Support
Certified
Availability
Cloud Enterprise
OpenTelemetry OTLP OTel

parquet

Reads and decodes Parquet files into a stream of structured messages.

Support
Certified Community

pg_stream

Streams changes from a PostgreSQL database using logical replication.

Type
Input
Support
Certified
Availability
Cloud Enterprise

postgres_cdc

Streams changes from a PostgreSQL database using logical replication.

Type
Support
Certified
Availability
Cloud Enterprise

pulsar

Reads messages from an Apache Pulsar server.

Type
Support
Community
Apache Pulsar

read_until

Reads messages from a child input until a consumed message passes a Bloblang query, at which point the input closes.

Type
Support
Certified
Availability

redis_list

Pops messages from the beginning of a Redis list using the BLPop command.

Type
Support
Certified
Availability
Redis List Redis Lists Redis

redis_pubsub

Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands.

Type
Support
Certified
Availability
Redis PubSub Redis Pub/Sub Redis

redis_scan

Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands.

Type
Support
Certified
Availability
Redis

redis_streams

Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The client_id should be unique for each consumer of a group.

Type
Support
Certified
Availability
Redis Streams Redis

redpanda_common

Consumes data from a Redpanda (Kafka) broker, using credentials defined in a common top-level redpanda config block.

Type
Support
Certified
Availability
Cloud Enterprise

redpanda_migrator

Consumes records from a source Kafka or Redpanda cluster for the redpanda_migrator output to migrate.

Type
Support
Certified
Availability

resource

Resource is an input type that channels messages from a resource input, identified by its name.

Support
Certified
Availability

salesforce

Executes a Salesforce SOQL query once and emits a message for each record.

Type
Support
Certified
Availability
Cloud Enterprise

salesforce_cdc

Streams Salesforce Change Data Capture (CDC) and Platform Events from the Pub/Sub gRPC API, optionally preceded by a REST snapshot of the...

Type
Support
Certified
Availability
Cloud Enterprise
Salesforce Salesforce CDC

salesforce_graphql

Executes a Salesforce GraphQL (UIAPI) query once and emits a message for each record.

Type
Support
Certified
Availability
Cloud Enterprise
Salesforce Salesforce GraphQL

sap_hana

Reads rows from a SAP HANA table.

Type
Support
Certified
Availability
Cloud Enterprise
SAP HANA

schema_registry

Reads schemas from a schema registry.

Type
Support
Certified
Availability

sequence

Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next,...

Type
Support
Certified
Availability

sftp

Consumes files from an SFTP server.

Type
Support
Certified
Availability

slack

Connects to Slack using Socket Mode and receives events, interactions, and slash commands.

Type
Support
Certified
Availability
Cloud Enterprise

slack_users

Returns the full profile of all users in your Slack organization using the users.list API method. You can filter the returned users by team ID.

Type
Support
Certified
Availability
Cloud Enterprise
Slack Users

socket

Connects to a tcp or unix socket and consumes a continuous stream of messages.

Type
Support
Certified

socket_server

Creates a server that receives a stream of messages over a TCP, UDP or Unix socket.

Type
Support
Certified

spicedb_watch

Consume messages from the Watch API from SpiceDB.

Type
Support
Community
Availability

splunk

Consumes messages from Splunk.

Type
Support
Certified
Availability
Cloud Enterprise

sql_raw

Executes a select query and creates a message for each row received.

Support
Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino
Availability
SQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino

sql_select

Executes a select query and creates a message for each row received.

Support
Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino
Availability
SQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino

stdin

Consumes data piped to stdin, chopping it into individual messages according to the specified scanner.

Type
Support
Certified

subprocess

Executes a command, runs it as a subprocess, and consumes messages from it over stdout.

Support
Community

tigerbeetle_cdc

Streams change data capture (CDC) events from a TigerBeetle cluster.

Type
Support
Certified

timeplus

Executes a query on Timeplus Enterprise and creates a message from each row received.

Type
Support
Community
Availability

twitter_search

Consumes tweets matching a given search using the Twitter recent search V2 API.

Type
Support
Community

websocket

Connects to a websocket server and continuously receives messages.

Type
Support
Certified
Availability

zmq4

Consumes messages from a ZeroMQ socket.

Type
Support
Community

aws_cloudwatch

Send metrics to AWS CloudWatch using the PutMetricData endpoint.

Type
Support
Community

influxdb

Send metrics to InfluxDB 1.x using the /write endpoint.

Type
Support
Community

json_api

Serves metrics as JSON object with the service wide HTTP service at the endpoints /stats and /metrics.

Type
Support
Certified

logger

Prints aggregated metrics through the logger.

Type
Support
Certified

open_telemetry_collector

Sends metrics to one or more OpenTelemetry Collector endpoints.

Type
Support
Certified
Availability
Cloud Enterprise
OpenTelemetry

prometheus

Host endpoints (/metrics and /stats) for Prometheus scraping.

Type
Support
Certified
Availability

statsd

Pushes metrics using the StatsD protocol. Supported tagging formats are 'none', 'datadog' and 'influxdb'.

Type
Support
Certified

arc

Writes data to an Arc database via the msgpack ingestion endpoint.

Type
Support
Community
Availability

aws_kinesis_firehose

Sends messages to a Kinesis Firehose delivery stream.

Type
Support
Certified
Availability
AWS Kinesis Firehose Amazon Kinesis Firehose Kinesis Firehose

aws_sns

Sends messages to an AWS SNS topic.

Type
Support
Community
Availability
AWS SNS Amazon SNS SNS Simple Notification Service

azure_data_lake_gen2

Sends message parts as files to an Azure Data Lake Gen2 filesystem. Each file is uploaded with the filename specified with the path field.

Type
Support
Certified
Availability
Microsoft Azure Azure

cache

Stores each message in a cache.

Support
Certified
Availability

cyborgdb

Inserts items into a CyborgDB encrypted vector index.

Type
Support
Community
Availability

cypher

Writes batches of messages to any database that speaks the Bolt protocol, such as Neo4j or Memgraph, using operations written in the Cypher query...

Type
Support
Community

doris_stream_load

Writes batches of messages into Apache Doris using Stream Load.

Type
Support
Community

drop

Drops all messages.

Type
Support
Certified
Availability

drop_on

Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked)...

Type
Support
Certified
Availability

elasticsearch_v8

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

Type
Support
Certified
Availability

elasticsearch_v9

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

Type
Support
Community

fallback

Attempts to send each message to a child output, starting from the first output on the list.

Type
Support
Certified
Availability

gcp_bigquery

Sends messages as new rows to a Google Cloud BigQuery table.

Type
Support
Certified
Availability
GCP BigQuery Google BigQuery BigQuery

gcp_bigquery_write_api

Streams data into BigQuery using the Storage Write API.

Type
Support
Certified
Availability
Cloud Enterprise
GCP BigQuery

iceberg

Writes data to Apache Iceberg tables using the REST catalog API.

Type
Support
Certified
Availability
Cloud Enterprise
Apache Iceberg Apache Polaris AWS Glue Databricks Unity Catalog

opensearch

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

Type
Support
Certified
Availability

pinecone

Inserts items into a Pinecone index.

Type
Support
Certified
Availability

pusher

Publishes messages to the Pusher API.

Type
Support
Community

qdrant

Adds items to a Qdrant collection.

Support
Certified
Availability

questdb

Pushes messages to a QuestDB table.

Type
Support
Certified
Availability

redis_hash

Sets Redis hash objects using the HSET command.

Type
Support
Certified
Availability
Redis Hash Redis

reject

Rejects all messages, treating them as though the output destination failed to publish them.

Type
Support
Certified
Availability

reject_errored

Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output.

Type
Support
Certified
Availability

retry

Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries...

Support
Certified
Availability

salesforce_sink

Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration.

Type
Support
Certified
Availability
Cloud Enterprise
Salesforce Salesforce Sink

slack_post

Posts a new message to a Slack channel using the Slack API method chat.postMessage.

Type
Support
Certified
Availability
Cloud Enterprise
Slack Post

slack_reaction

Adds or removes an emoji reaction on a Slack message.

Type
Support
Certified
Availability
Cloud Enterprise
Slack Reaction

snowflake_put

Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables.

Type
Support
Certified
Availability
Cloud Enterprise
Snowflake

snowflake_streaming

Ingests data into Snowflake using the Snowpipe Streaming classic architecture.

Type
Support
Certified
Availability
Cloud Enterprise
Snowflake Streaming

snowflake_streaming_pipe

Ingest data into Snowflake using Snowpipe Streaming, writing through a Snowflake pipe.

Type
Support
Certified
Availability
Cloud Enterprise
Snowflake Streaming Pipe

splunk_hec

Publishes messages to a Splunk HTTP Endpoint Collector (HEC).

Type
Support
Certified
Availability
Cloud Enterprise
Splunk

sql_insert

Inserts a row into an SQL database for each message.

Support
Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino
Availability
SQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino

stdout

Prints messages to stdout as a continuous stream of data.

Type
Support
Certified

switch

The switch output type allows you to route messages to different outputs based on their contents.

Support
Certified
Availability

sync_response

Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type.

Support
Certified
Availability

a2a_message

Sends messages to an A2A (Agent-to-Agent) protocol agent and returns the response.

Type
Support
Certified
Availability
Cloud Enterprise

archive

Archives all the messages of a batch into a single message according to the selected archive format.

Type
Support
Certified
Availability
ZIP TAR GZIP

avro

Performs Avro based operations on messages based on a schema.

Support
Community
Availability

awk

Executes an AWK program on messages.

Type
Support
Community

aws_bedrock_chat

Generates responses to messages in a chat conversation, using the AWS Bedrock API.

Type
Support
Certified
Availability
Amazon AWS Bedrock Chat

aws_bedrock_embeddings

Computes vector embeddings on text, using the AWS Bedrock API.

Type
Support
Certified
Availability
Amazon AWS Bedrock Embeddings

aws_dynamodb_partiql

Executes a PartiQL expression against a DynamoDB table for each message.

Type
Support
Certified
Availability
Amazon AWS DynamoDB PartiQL

aws_lambda

Invokes an AWS lambda for each message.

Type
Support
Certified
Availability
AWS Lambda Amazon Lambda Lambda

benchmark

Logs basic throughput statistics of messages that pass through this processor.

Type
Support
Certified
Availability

bloblang

Executes a Bloblang mapping on messages.

Type
Support
Certified
Availability

bounds_check

Removes messages (and batches) that do not fit within certain size boundaries.

Type
Support
Certified
Availability

branch

The branch processor allows you to create a new request message via a Bloblang mapping, execute a list of processors on the request messages, and,...

Type
Support
Certified
Availability

cached

Cache the result of applying one or more processors to messages identified by a key.

Type
Support
Certified
Availability

catch

Applies a list of child processors only when a previous processing step has failed.

Type
Support
Certified
Availability

cohere_chat

Generates responses to messages in a chat conversation, using the Cohere API.

Type
Support
Certified
Availability

cohere_embeddings

Generates vector embeddings to represent input text, using the Cohere API.

Type
Support
Certified
Availability

cohere_rerank

Reranks a list of documents by their relevance to a query, using the Cohere API.

Type
Support
Certified
Availability

command

Executes a command for each message.

Type
Support
Certified

compress

Compresses messages according to the selected algorithm. Supported compression algorithms are: [flate gzip lz4 pgzip snappy zlib]

Type
Support
Certified
Availability

crash

Crashes the process using a fatal log message.

Type
Support
Certified

decompress

Decompresses messages according to the selected algorithm. Supported decompression algorithms are: [bzip2 flate gzip lz4 pgzip snappy zlib]

Support
Certified
Availability

dedupe

Deduplicates messages by storing a key value in a cache using the add operator. If the key already exists within the cache it is dropped.

Type
Support
Certified
Availability

ffi

Invokes functions from shared libraries using a Foreign Function Interface (FFI).

Type
Support
Certified
Foreign Function Interface

for_each

A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message.

Type
Support
Certified
Availability

gcp_vertex_ai_chat

Generates responses to messages in a chat conversation, using the Vertex AI API.

Type
Support
Certified
Availability
GCP Vertex AI Google Cloud GCP

gcp_vertex_ai_embeddings

Generates vector embeddings to represent input text, using the Vertex AI API.

Type
Support
Certified
Availability
Google Cloud GCP

google_drive_download

Downloads files from Google Drive.

Type
Support
Certified
Availability
Cloud Enterprise

google_drive_list_labels

Lists labels for a file in Google Drive.

Type
Support
Certified
Availability
Cloud Enterprise

google_drive_search

Searches Google Drive for files matching the provided query.

Type
Support
Certified
Availability
Cloud Enterprise

grok

Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value...

Type
Support
Community

group_by

Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a Bloblang query.

Type
Support
Certified
Availability

group_by_value

Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a function interpolated string...

Type
Support
Certified
Availability

http

Performs an HTTP request using a message batch as the request body, and replaces the original message parts with the body of the response.

Type
Support
Certified
Availability

insert_part

Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end.

Type
Support
Certified
Availability

javascript

Executes a provided JavaScript code block or file for each message.

Type
Support
Certified

jmespath

Executes a JMESPath query on JSON documents and replaces the message with the resulting document.

Type
Support
Certified
Availability

jq

Transforms and filters messages using jq queries.

Type
Support
Certified
Availability

json_schema

Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances.

Type
Support
Certified
Availability
JSON Schema

log

Prints a log event for each message.

Type
Support
Certified
Availability

mapping

Executes a Bloblang mapping on messages, creating a new document that replaces (or filters) the original message.

Type
Support
Certified
Availability

metric

Emit custom metrics by extracting values from messages.

Type
Support
Certified
Availability

msgpack

Converts messages to or from the MessagePack format.

Type
Support
Community

mutation

Executes a Bloblang mapping and directly transforms the contents of messages, mutating (or deleting) them.

Type
Support
Certified
Availability

nats_request_reply

Sends a message to a NATS subject and expects a reply back from a NATS subscriber acting as a responder.

Type
Support
Certified
Availability
NATS Request Reply

ollama_chat

Generates responses to messages in a chat conversation, using the Ollama API.

Type
Support
Certified

ollama_embeddings

Generates vector embeddings from text, using the Ollama API.

Type
Support
Certified

ollama_moderation

Checks whether responses from a large language model (LLM) are safe, using the Ollama API.

Type
Support
Certified

openai_chat_completion

Generates responses to messages in a chat conversation, using the OpenAI API.

Type
Support
Certified
Availability

openai_embeddings

Generates vector embeddings to represent input text, using the OpenAI API.

Type
Support
Certified
Availability

openai_image_generation

Generates an image from a text description and other attributes, using OpenAI API.

Type
Support
Certified
Availability

openai_speech

Generates audio from a text description and other attributes, using OpenAI API.

Type
Support
Certified
Availability

openai_transcription

Generates a transcription of spoken audio in the input language, using the OpenAI API.

Type
Support
Certified
Availability

openai_translation

Translates spoken audio into English, using the OpenAI API.

Type
Support
Certified
Availability

parallel

A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the...

Type
Support
Certified
Availability

parquet_decode

Decodes Parquet files into a batch of structured messages.

Type
Support
Certified
Availability

parquet_encode

Encodes Parquet files from a batch of structured messages.

Type
Support
Certified
Availability

parse_log

Parses common log formats into structured data. This is easier and often much faster than grok.

Type
Support
Community
Availability

processors

A processor grouping several sub-processors.

Type
Support
Certified
Availability

protobuf

Performs conversions to or from a protobuf message.

Type
Support
Certified

rate_limit

Throttles the throughput of a pipeline according to a specified rate_limit resource.

Type
Support
Certified
Availability

redis_script

Performs actions against Redis using LUA scripts.

Type
Support
Certified
Availability
Redis Script

redpanda_data_transform

Executes a Redpanda Data Transform as a processor.

Type
Support
Certified

schema_registry_decode

Automatically decodes and validates messages with schemas from a Confluent Schema Registry service.

Type
Support
Certified
Availability

schema_registry_encode

Automatically encodes and validates messages with schemas from a Confluent Schema Registry service.

Type
Support
Certified
Availability

select_parts

Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored.

Type
Support
Certified
Availability

sentry_capture

Captures log events from messages and submits them to Sentry.

Type
Support
Community

slack_thread

Reads a Slack thread using the Slack API method conversations.replies.

Type
Support
Certified
Availability
Cloud Enterprise
Slack Thread

sleep

Sleep for a period of time specified as a duration string for each message.

Type
Support
Certified
Availability

split

Breaks message batches (synonymous with multiple part messages) into smaller batches.

Type
Support
Certified
Availability

string_split

Splits a string by a delimiter into an array.

Type
Support
Certified
Availability

text_chunker

Breaks text-based message content into chunks using a configurable strategy, typically to create vector embeddings of large documents.

Type
Support
Certified
Availability

try

Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared).

Type
Support
Certified
Availability

try_catch

Executes a list of child processors on each message and, if any of them fail, executes a separate list of catch processors to recover from or...

Type
Support
Community
Availability

unarchive

Unarchives messages according to the selected archive format into multiple messages within a batch.

Type
Support
Certified
Availability
ZIP TAR GZIP Archive

wasm

Executes a function exported by a WASM module for each message.

Type
Support
Community

while

A processor that checks a Bloblang query against each batch of messages and executes child processors on them for as long as the query resolves to...

Type
Support
Certified
Availability

workflow

Executes a topology of branch processors, performing them in parallel where possible.

Type
Support
Certified
Availability

xml

Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value.

Type
Support
Community
Availability

local

A simple X every Y rate limit that can be shared across any number of components within a pipeline.

Type
Support
Certified
Availability

chunker

Split an input stream into chunks of a given number of bytes.

Type
Support
Certified
Availability

json_array

Consumes a stream of one or more JSON elements within a top level array.

Type
Support
Community
Availability

json_documents

Consumes a stream of one or more JSON documents.

Type
Support
Certified
Availability

lines

Split an input stream into a message per line of data.

Type
Support
Certified
Availability

re_match

Split an input stream into segments matching against a regular expression.

Type
Support
Certified
Availability

skip_bom

Skip one or more byte order marks for each opened child scanner.

Type
Support
Certified
Availability

tar

Consume a tar archive file by file.

Type
Support
Certified
Availability

to_the_end

Read the input stream all the way until the end and deliver it as a single message.

Type
Support
Certified
Availability

gcp_cloudtrace

Send tracing events to a Google Cloud Trace.

Type
Support
Certified
Availability
GCP Cloud Trace

jaeger

Send tracing events to a Jaeger agent or collector.

Type
Support
Community

About Components

Every Redpanda Connect pipeline has at least one input, an optional buffer, an output and any number of processors:

input:
  kafka:
    addresses: [ TODO ]
    topics: [ foo, bar ]
    consumer_group: foogroup

buffer:
  type: none

pipeline:
  processors:
  - mapping: |
      message = this
      meta.link_count = links.length()

output:
  aws_s3:
    bucket: TODO
    path: '${! meta("kafka_topic") }/${! json("message.id") }.json'

These are the main components within Redpanda Connect and they provide the majority of useful behavior.

Observability components

There are also the observability components: http, logger, metrics, and tracing, which allow you to specify how Redpanda Connect exposes observability data.

http:
  address: 0.0.0.0:4195
  enabled: true
  debug_endpoints: false

logger:
  format: json
  level: WARN

metrics:
  statsd:
    address: localhost:8125
    flush_period: 100ms

tracer:
  jaeger:
    agent_address: localhost:6831

Resource components

Finally, there are caches and rate limits. These are components that are referenced by core components and can be shared.

input:
  http_client: # This is an input
    url: TODO
    rate_limit: foo_ratelimit # This is a reference to a rate limit

pipeline:
  processors:
    - cache: # This is a processor
        resource: baz_cache # This is a reference to a cache
        operator: add
        key: '${! json("id") }'
        value: "x"
    - mapping: root = if errored() { deleted() }

rate_limit_resources:
  - label: foo_ratelimit
    local:
      count: 500
      interval: 1s

cache_resources:
  - label: baz_cache
    memcached:
      addresses: [ localhost:11211 ]

It’s also possible to configure inputs, outputs and processors as resources which allows them to be reused throughout a configuration with the resource input, resource output and resource processor respectively.

For more information about any of these component types check out their sections:

Support levels

Each connector has a corresponding level of support, which determines whether it is suitable for production environments or community-driven projects.

  • Certified: Production-ready and available to purchase with Redpanda support.

  • Community: Not considered production-ready and available only for community support in the Redpanda community on GitHub.

Anyone can use the Certified and Community connectors. However, to use Enterprise connectors, you need an Enterprise Edition license.