Migrating to Version 2

Redpanda Connect version 2 comes with a small number of backwards incompatible changes that are organized into three sections; configuration, service and Go API.

These changes are considered minor and therefore upgrading is not expected to cause significant problems for any users.

Configuration

Type inference

Version 2 comes with the ability to infer the type of components in configuration files whenever the field is omitted. You can read more about this behavior here.

This feature is not expected to impact the vast majority of users. However, there is one exception where a malformed section containing unused type parameters but a missing type field will be interpreted differently. For example, the following config:

pipeline:
  processors:
  - # type: text
    text:
      operator: set
      value: "delete all your content"

In V1 would be interpreted as a bounds_check processor as it is the default processor type, whereas V2 would infer this to be a text processor based on its fields.

Migration guide

Most users should not be impacted by this change, and a config file that is vulnerable to the regression would report linting errors in V1.

You can quickly verify that your configs are interpreted without regression by comparing the output of benthos -c ./yourconfig.yaml --print-yaml with V1 and V2. If they are the same then you are not affected.

Field default value changes

In version 2 the field unsubscribe_on_close of the nats_stream input is now false by default.

Service

The recommended way to create plugins for Redpanda Connect is outlined in this repository. Therefore the following experimental plugin related flags have been removed from the service: swap-envs, plugins-dir, list-input-plugins, list-output-plugins, list-processor-plugins, list-condition-plugins.

The flag swap-envs has also been removed for clarity, as it had no impact on JSON reference resolution. If this flag is being used please open an issue and it can be reimplemented to be fully compliant.

Go API

Condition package moved

The package github.com/Jeffail/benthos/lib/processor/condition has been changed to github.com/Jeffail/benthos/lib/condition. Migrating should be a simple case of applying a find/replace on your codebase:

find . -name "*.go" | \
  xargs sed -i 's/benthos\/lib\/processor\/condition/benthos\/lib\/condition/g'

Interface changes

The following interface changes have occurred to core Redpanda Connect components:

  • types.Cache now has types.Closable embedded.

  • types.RateLimit now has types.Closable embedded.

  • types.Manager has new method GetPlugin.

  • log.Modular has new method WithFields.