Components Catalog

Stores consumed messages in memory and acknowledges them at the input level.

Do not buffer messages. This is the default and most resilient configuration.

Chops a stream of messages into tumbling or sliding windows of fixed temporal size, following the system clock.

Stores key/value pairs as a single document in a DynamoDB table.

Stores each item in an S3 bucket as a file, where an item ID is the path of the item within the bucket.

Use a Google Cloud Storage bucket as a cache.

Stores key/value pairs in a lru in-memory cache. This cache is therefore reset every time the service restarts.

Connects to a cluster of memcached services, a prefix can be specified to allow multiple cache types to share a memcached cluster under different...

Use a MongoDB instance as a cache.

Combines multiple caches as levels, performing read-through and write-through operations across them.

Caches key/value pairs in a NATS key-value bucket.

Noop is a cache that stores nothing, all gets returns not found. Why? Sometimes doing nothing is the braver option.

Use a Redis instance as a cache. The expiration can be set to zero or an empty string in order to set no expiration.

A Kafka cache implemented using the Franz Kafka client library.

Stores key/value pairs in a map held in the memory-bound Ristretto cache.

Uses an SQL database table as a destination for storing cache key/value items.

Stores key/value pairs in a ttlru in-memory cache. This cache is therefore reset every time the service restarts.

Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ.

Consumes log events from AWS CloudWatch Logs.

Reads change data capture (CDC) events from DynamoDB Streams.

Receive messages from one or more Kinesis streams.

Consume messages from an AWS SQS URL.

Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix.

Executes a SQL query against Azure CosmosDB and creates a batch of messages from each page of items.

Dequeue objects from an Azure Storage Queue.

Queries an Azure Storage Account Table, optionally with multiple filters.

Consumes data from a child input and applies a batching policy to the stream.

Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel.

Reads one or more CSV files as structured records following the format described in RFC 4180.

Receive messages delivered over HTTP.

Executes a SELECT query against BigQuery and creates a message for each row received.

Consumes messages from a GCP Cloud Pub/Sub subscription.

Consumes change records from a Google Cloud Spanner change stream.

Generates messages at a given interval using a Bloblang mapping executed without a context.

Clones a Git repository, reads its contents, then polls for new commits at a configurable interval.

Connects to a server and continuously performs requests for a single message.

Receive messages POSTed over HTTP(S). HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified.

Directly connects to an output within the same process by a chosen ID, to link isolated streams when running in streams mode.

Streams Jira issues, comments, or changelog entries via JQL with incremental polling.

Connects to Kafka brokers and consumes one or more topics.

A Kafka input using the Franz Kafka client library.

Enables Change Data Capture by consuming from Microsoft SQL Server's change tables.

Streams changes from a MongoDB replica set.

Subscribe to topics on MQTT brokers.

Streams data changes from a MySQL or MariaDB database, using the binary log to capture data updates.

Subscribes to a NATS subject.

Reads messages from NATS JetStream subjects.

Enables Change Data Capture by consuming from OracleDB.

Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol.

Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol.

pg_stream

Streams changes from a PostgreSQL database using logical replication.

Streams changes from a PostgreSQL database using logical replication.

Reads messages from a child input until a consumed message passes a Bloblang query, at which point the input closes.

Pops messages from the beginning of a Redis list using the BLPop command.

Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands.

Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands.

Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The client_id should be unique for each consumer of a group.

Consumes data from a Redpanda (Kafka) broker, using credentials defined in a common top-level redpanda config block.

Consumes records from a source Kafka or Redpanda cluster for the redpanda_migrator output to migrate.

Resource is an input type that channels messages from a resource input, identified by its name.

Executes a Salesforce SOQL query once and emits a message for each record.

Streams Salesforce Change Data Capture (CDC) and Platform Events from the Pub/Sub gRPC API, optionally preceded by a REST snapshot of the...

Executes a Salesforce GraphQL (UIAPI) query once and emits a message for each record.

Reads rows from a SAP HANA table.

Reads schemas from a schema registry.

Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next,...

Consumes files from an SFTP server.

Connects to Slack using Socket Mode and receives events, interactions, and slash commands.

Returns the full profile of all users in your Slack organization using the users.list API method. You can filter the returned users by team ID.

Consume messages from the Watch API from SpiceDB.

Consumes messages from Splunk.

Executes a select query and creates a message for each row received.

Executes a select query and creates a message for each row received.

Executes a query on Timeplus Enterprise and creates a message from each row received.

Connects to a websocket server and continuously receives messages.

Sends metrics to one or more OpenTelemetry Collector endpoints.

Host endpoints (/metrics and /stats) for Prometheus scraping.

Writes data to an Arc database via the msgpack ingestion endpoint.

Sends messages to a Kinesis Firehose delivery stream.

Sends messages to an AWS SNS topic.

Sends message parts as files to an Azure Data Lake Gen2 filesystem. Each file is uploaded with the filename specified with the path field.

Stores each message in a cache.

Inserts items into a CyborgDB encrypted vector index.

Drops all messages.

Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked)...

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

Attempts to send each message to a child output, starting from the first output on the list.

Sends messages as new rows to a Google Cloud BigQuery table.

Streams data into BigQuery using the Storage Write API.

Writes data to Apache Iceberg tables using the REST catalog API.

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

Inserts items into a Pinecone index.

Adds items to a Qdrant collection.

Pushes messages to a QuestDB table.

Sets Redis hash objects using the HSET command.

Rejects all messages, treating them as though the output destination failed to publish them.

Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output.

Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries...

Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration.

Posts a new message to a Slack channel using the Slack API method chat.postMessage.

Adds or removes an emoji reaction on a Slack message.

Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables.

Ingests data into Snowflake using the Snowpipe Streaming classic architecture.

Ingest data into Snowflake using Snowpipe Streaming, writing through a Snowflake pipe.

Publishes messages to a Splunk HTTP Endpoint Collector (HEC).

Inserts a row into an SQL database for each message.

The switch output type allows you to route messages to different outputs based on their contents.

Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type.

Sends messages to an A2A (Agent-to-Agent) protocol agent and returns the response.

Archives all the messages of a batch into a single message according to the selected archive format.

Performs Avro based operations on messages based on a schema.

Generates responses to messages in a chat conversation, using the AWS Bedrock API.

Computes vector embeddings on text, using the AWS Bedrock API.

Executes a PartiQL expression against a DynamoDB table for each message.

Invokes an AWS lambda for each message.

Logs basic throughput statistics of messages that pass through this processor.

Executes a Bloblang mapping on messages.

Removes messages (and batches) that do not fit within certain size boundaries.

The branch processor allows you to create a new request message via a Bloblang mapping, execute a list of processors on the request messages, and,...

Cache the result of applying one or more processors to messages identified by a key.

Applies a list of child processors only when a previous processing step has failed.

Generates responses to messages in a chat conversation, using the Cohere API.

Generates vector embeddings to represent input text, using the Cohere API.

Reranks a list of documents by their relevance to a query, using the Cohere API.

Compresses messages according to the selected algorithm. Supported compression algorithms are: [flate gzip lz4 pgzip snappy zlib]

Decompresses messages according to the selected algorithm. Supported decompression algorithms are: [bzip2 flate gzip lz4 pgzip snappy zlib]

Deduplicates messages by storing a key value in a cache using the add operator. If the key already exists within the cache it is dropped.

A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message.

Generates responses to messages in a chat conversation, using the Vertex AI API.

Generates vector embeddings to represent input text, using the Vertex AI API.

Downloads files from Google Drive.

Lists labels for a file in Google Drive.

Searches Google Drive for files matching the provided query.

Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a Bloblang query.

Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a function interpolated string...

Performs an HTTP request using a message batch as the request body, and replaces the original message parts with the body of the response.

Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end.

Executes a JMESPath query on JSON documents and replaces the message with the resulting document.

Transforms and filters messages using jq queries.

Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances.

Prints a log event for each message.

Executes a Bloblang mapping on messages, creating a new document that replaces (or filters) the original message.

Emit custom metrics by extracting values from messages.

Executes a Bloblang mapping and directly transforms the contents of messages, mutating (or deleting) them.

Sends a message to a NATS subject and expects a reply back from a NATS subscriber acting as a responder.

Generates responses to messages in a chat conversation, using the Ollama API.

Generates vector embeddings from text, using the Ollama API.

Checks whether responses from a large language model (LLM) are safe, using the Ollama API.

Generates responses to messages in a chat conversation, using the OpenAI API.

Generates vector embeddings to represent input text, using the OpenAI API.

Generates an image from a text description and other attributes, using OpenAI API.

Generates audio from a text description and other attributes, using OpenAI API.

Generates a transcription of spoken audio in the input language, using the OpenAI API.

Translates spoken audio into English, using the OpenAI API.

A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the...

Decodes Parquet files into a batch of structured messages.

Encodes Parquet files from a batch of structured messages.

Parses common log formats into structured data. This is easier and often much faster than grok.

A processor grouping several sub-processors.

Throttles the throughput of a pipeline according to a specified rate_limit resource.

Performs actions against Redis using LUA scripts.

Automatically decodes and validates messages with schemas from a Confluent Schema Registry service.

Automatically encodes and validates messages with schemas from a Confluent Schema Registry service.

Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored.

Reads a Slack thread using the Slack API method conversations.replies.

Sleep for a period of time specified as a duration string for each message.

Breaks message batches (synonymous with multiple part messages) into smaller batches.

Splits a string by a delimiter into an array.

Breaks text-based message content into chunks using a configurable strategy, typically to create vector embeddings of large documents.

Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared).

Executes a list of child processors on each message and, if any of them fail, executes a separate list of catch processors to recover from or...

Unarchives messages according to the selected archive format into multiple messages within a batch.

A processor that checks a Bloblang query against each batch of messages and executes child processors on them for as long as the query resolves to...

Executes a topology of branch processors, performing them in parallel where possible.

Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value.

A simple X every Y rate limit that can be shared across any number of components within a pipeline.

Split an input stream into chunks of a given number of bytes.

Consumes a stream of one or more JSON elements within a top level array.

Consumes a stream of one or more JSON documents.

Split an input stream into a message per line of data.

Split an input stream into segments matching against a regular expression.

Skip one or more byte order marks for each opened child scanner.

Consume a tar archive file by file.

Read the input stream all the way until the end and deliver it as a single message.

Send tracing events to a Google Cloud Trace.