# Plugins

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [connect-full.txt](https://docs.redpanda.com/connect-full.txt)

---
title: Plugins
latest-connect-version: 4.93.0
latest-operator-version: v26.1.4
latest-console-tag: v3.7.3
latest-redpanda-tag: v26.1.9
docname: about
page-component-name: connect
page-version: master
page-component-version: master
page-component-title: Connect
page-relative-src-path: about.adoc
page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/plugins/pages/about.adoc
description: Create and load dynamic plugins at runtime with gRPC.
page-git-created-date: "2025-06-11"
page-git-modified-date: "2025-06-11"
---

<!-- Source: https://docs.redpanda.com/connect/plugins/about.md -->

Redpanda Connect’s RPC plugin framework is a powerful Apache 2.0 licensed feature that takes data streaming flexibility to the next level. By enabling you to create and load plugins at runtime using any programming language that supports gRPC, this framework opens up a world of new integration possibilities beyond Go. Whether you need to leverage specialized libraries, integrate with AI/ML tools, or build custom components in your preferred language, Redpanda Connect plugins provide the extensibility and interoperability to make it happen.

## [](#plugin-types)Plugin types

Redpanda Connect supports two types of plugins:

Compiled plugins

Built in Go and compiled directly into the Redpanda Connect binary. These offer maximum performance for critical workloads.

Dynamic plugins

External executables that communicate with Redpanda Connect via gRPC. These provide language flexibility and can be loaded at runtime without rebuilding the binary.

This documentation focuses on dynamic plugins.

## [](#when-to-use-dynamic-plugins)When to use dynamic plugins

Dynamic plugins are ideal when you need to extend Redpanda Connect’s capabilities without the constraints of Go. Use cases include:

-   Use existing libraries not available in Go

-   Write plugins in languages other than Go (such as Python)

-   Add plugins without rebuilding the entire Redpanda Connect binary

-   Deploy plugins independently of your main Redpanda Connect deployment

-   Integrate with AI/ML frameworks, data science libraries, or specialized tools


## [](#architecture)Architecture

Dynamic plugins run as separate processes and communicate with the main Redpanda Connect process through gRPC over Unix sockets. This architecture provides the following benefits:

-   Process isolation: Plugin crashes don’t affect the main engine.

-   Language agnostic: Support for any language with gRPC libraries.

-   Modular design: Each plugin maps to a single subprocess.

-   Battle-tested communication. Uses proven gRPC technology.


The system includes three compiled plugin wrappers (one for each component type: `BatchInput`, `BatchProcessor`, and `BatchOutput`) that handle the communication with external plugin executables.

## [](#supported-languages)Supported languages

Redpanda Connect currently provides the following SDKs:

-   **Go**: Familiar environment for existing Redpanda Connect developers with type-safe interfaces

-   **Python**: Access to rich ecosystem including PyTorch, TensorFlow, Hugging Face Transformers, LangChain, Pandas, and NumPy


## [](#performance-considerations)Performance considerations

Dynamic plugins introduce some serialization and IPC overhead compared to compiled plugins. To minimize performance impact:

-   The system uses batch components exclusively to amortize cross-process communication costs.

-   Each plugin runs in a separate process with predictable resource usage.

-   For maximum performance in critical workloads, use compiled plugins instead.


## [](#prerequisites)Prerequisites

Before creating dynamic plugins, ensure you have:

-   Redpanda Connect v4.56.0 or later

-   Python environment (for Python plugins)

-   Required language SDKs installed


### [](#install-redpanda-connect)Install Redpanda Connect

Verify your version, and upgrade if necessary:

```shell
rpk --version
rpk connect upgrade
```

## [](#create-your-first-plugin)Create your first plugin

This example demonstrates creating a simple FizzBuzz processor plugin in Python.

### [](#step-1-set-up-the-environment)Step 1: Set up the environment

```shell
# Create a new directory for your plugin
mkdir fizzbuzz_plugin
cd fizzbuzz_plugin

# Initialize with link:https://docs.astral.sh/uv/[uv] (or use pip/conda)
uv init .
uv add redpanda_connect
```

### [](#step-2-download-the-example)Step 2: Download the example

```shell
# Download the example processor
curl -o fizzbuzz_processor.py \
  https://raw.githubusercontent.com/redpanda-data/connect/main/public/plugin/python/examples/fizzbuzz_processor.py
```

### [](#step-3-create-the-plugin-configuration)Step 3: Create the plugin configuration

Create `plugin.yaml`:

```yaml
name: fizzbuzz
summary: Transforms numbers according to FizzBuzz rules (3=Fizz, 5=Buzz, 15=FizzBuzz)
command: ["uv", "run", "fizzbuzz_processor.py"]
type: processor
fields: []
```

### [](#step-4-create-the-pipeline-configuration)Step 4: Create the pipeline configuration

Create `connect.yaml`:

```yaml
input:
  generate:
    interval: 1s
    mapping: |
      root = counter() % 20 + 1 # Generate numbers 1-20

pipeline:
  processors:
  - fizzbuzz: {}

output:
  stdout:
    codec: lines
```

### [](#step-5-run-the-plugin)Step 5: Run the plugin

Execute the pipeline with your dynamic plugin:

```shell
rpk connect run --rpc-plugins=plugin.yaml connect.yaml
```

Expected output:

```text
1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16
17
Fizz
19
Buzz
```

## [](#python-plugin-development)Python plugin development

### [](#basic-processor-structure)Basic processor structure

Python processors use the `@redpanda_connect.processor` decorator:

```python
import asyncio
import logging
import redpanda_connect

@redpanda_connect.processor
def transform_message(msg: redpanda_connect.Message) -> redpanda_connect.Message:
    # Your transformation logic here
    msg.payload = msg.payload.upper()
    return msg

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(redpanda_connect.processor_main(transform_message))
```

### [](#plugin-configuration-structure)Plugin configuration structure

Each plugin requires a configuration file with the following structure:

```yaml
name: <plugin-name>
summary: <brief-description>
command: ["<executable>", "<args>"]
type: <processor|input|output>
fields: []  # Configuration fields (if any)
```

### [](#message-handling)Message handling

The Python SDK provides access to message properties:

-   `msg.payload` - Message content

-   `msg.metadata` - Message metadata

-   Standard message transformation methods


## [](#next-steps)Next steps

-   Explore the [example plugins](https://github.com/redpanda-data/connect/tree/main/public/plugin/python/examples) in the Redpanda Connect repository

-   Join the [Redpanda Community Slack](https://redpanda.com/slack) to discuss plugin development

-   Review the [Redpanda Connect source code](https://github.com/redpanda-data/connect) for advanced use cases


## [](#related-topics)Related topics

-   [Components overview](https://docs.redpanda.com/connect/components/about/)

-   [Configuration overview](https://docs.redpanda.com/connect/configuration/about/)

-   [Getting started with Redpanda Connect](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/)


## Suggested labs

-   [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/)
-   [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/)
-   [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/)
-   [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/)
-   [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/)

[Search all labs](https://docs.redpanda.com/labs)