ffi

Available in: Self-Managed

Invoke a function within a shared library as a processing step.

  • Common

  • Advanced

processors:
  label: ""
  ffi:
    library_path: "" # No default (required)
    function_name: "" # No default (required)
    args_mapping: "" # No default (required)
    signature:
      return:
        type: "" # No default (required)
      parameters:
        type: "" # No default (required)
        out: false
processors:
  label: ""
  ffi:
    library_path: "" # No default (required)
    function_name: "" # No default (required)
    args_mapping: "" # No default (required)
    signature:
      return:
        type: "" # No default (required)
      parameters:
        type: "" # No default (required)
        out: false

The ffi processor allows you to dynamically load shared libraries (.so, .dylib, or .dll files) and invoke their functions as a processing step. This enables integration with external C/C++ libraries without requiring custom Go code or rebuilding Redpanda Connect.

The processor uses dlopen (or platform equivalent) to load the specified library at runtime, looks up the function by name, and invokes it with arguments provided through Bloblang mapping. Function signatures are defined using the signature field, specifying parameter and return types.

Results are returned as an array, where the first element is the return value (if not void), followed by any output parameters in their declaration order.

Requirements

Ensure that your environment meets the following requirements:

Fields

args_mapping

The bloblang expression that returns an array of arguments to pass into the foreign function.

Type: string

# Examples:
args_mapping: root = [42, now().ts_unix_nano(), content()]

function_name

The name of the function to load from the shared library.

Type: string

# Examples:
function_name: MyExternCFunction

library_path

The path to the shared library (.so, .dylib or .dll) file to load dynamically.

Type: string

# Examples:
library_path: libbar.6.so

# ---

library_path: libfoo.dylib

signature

The signature of the function.

Type: object

signature.parameters[]

The parameters of the function.

Type: object

signature.parameters[].out

If the parameter provided is an 'out' parameter, meaning if the function mutates the value, and the resulting value should be returned. This is only valid for pointer types.

Type: bool

Default: false

signature.parameters[].type

The data type of the parameter.

Type: string

Option Summary

byte*

A pointer to a byte array is provided as an argument. Note this byte array cannot be referenced once the function returns. args_mapping must return a byte array or string type for this argument, and the parameter in C for this should be void*.

int32

A 32 bit signed integer is provided as an argument

int64

A 64 bit signed integer is provided as an argument

signature.return

The configuration for the function’s result.

Type: object

signature.return.type

The data type of function’s return value

Type: string

Option Summary

int32

A 32 bit signed integer is returned

int64

A 64 bit signed integer is returned

void

The function returns nothing

Examples

Call a libc function

This is an example of loading libc.so and calling a function on linux.

pipeline:
  processors:
    - ffi:
        library_path: libc.6.so
        function_name: memcmp
        args_mapping: 'root = ["foo", "bar", 3]'
        signature:
          return:
            type: int32
          parameters:
            - type: byte*
            - type: byte*
            - type: int64