subprocess

Type:

Available in: Self-Managed

Executes a command as a subprocess and, for each message, will pipe its contents to the stdin stream of the process followed by a newline.

  • Common

  • Advanced

# Common config fields, showing default values
label: ""
subprocess:
  name: cat # No default (required)
  args: []
# All config fields, showing default values
label: ""
subprocess:
  name: cat # No default (required)
  args: []
  max_buffer: 65536
  codec_send: lines
  codec_recv: lines

This processor keeps the subprocess alive and requires very specific behavior from the command executed. If you wish to simply execute a command for each message take a look at the command processor instead.

The subprocess must then either return a line over stdout or stderr. If a response is returned over stdout then its contents will replace the message. If a response is instead returned from stderr it will be logged and the message will continue unchanged and will be marked as failed.

Rather than separating data by a newline it’s possible to specify alternative codec_send and codec_recv values, which allow binary messages to be encoded for logical separation.

The execution environment of the subprocess is the same as the Redpanda Connect instance, including environment variables and the current working directory.

The field max_buffer defines the maximum response size able to be read from the subprocess. This value should be set significantly above the real expected maximum response size.

Subprocess requirements

It is required that subprocesses flush their stdout and stderr pipes for each line. Redpanda Connect will attempt to keep the process alive for as long as the pipeline is running. If the process exits early it will be restarted.

Messages containing line breaks

If a message contains line breaks each line of the message is piped to the subprocess and flushed, and a response is expected from the subprocess before another line is fed in.

Fields

name

The command to execute as a subprocess.

Type: string

# Examples

name: cat

name: sed

name: awk

args

A list of arguments to provide the command.

Type: array

Default: []

max_buffer

The maximum expected response size.

Type: int

Default: 65536

codec_send

Determines how messages written to the subprocess are encoded, which allows them to be logically separated.

Type: string

Default: "lines" Requires version 3.37.0 or newer

Options: lines , length_prefixed_uint32_be , netstring .

codec_recv

Determines how messages read from the subprocess are decoded, which allows them to be logically separated.

Type: string

Default: "lines" Requires version 3.37.0 or newer

Options: lines , length_prefixed_uint32_be , netstring .