Migrating to Version 3
Redpanda Connect version 3 comes with some breaking service and configuration changes as well as a few breaking API changes.
Service
Memory map file buffer removed
The long deprecated mmap_file
buffer has been removed. If you were still relying on this buffer implementation then please raise an issue.
Old metrics paths removed
The following undocumented metrics paths have been removed:
-
input.parts.count
-
input.read.success
-
input.read.error
-
input.send.success
-
input.send.error
-
input.ack.success
-
input.ack.error
-
output.running
-
output.parts.count
-
output.send.success
-
output.parts.send.success
-
output.send.error
All of these paths have remaining equivalents.
Configuration
Metrics prefix
The configuration field prefix
within metrics
has been moved from the root
of the config object to individual types. E.g. when using statsd
the field
metrics.prefix
should be replaced with metrics.statsd.prefix
.
JSON paths
Many components within Redpanda Connect use an unspecified "JSON dot path" syntax for querying and setting fields within JSON documents. The format of these paths has been formalized to make them clearer and more generally useful, but this potentially breaks your paths when they query against hierarchies that contain arrays.
The formal specification for v3 can be found in this document.
The following components are affected:
-
awk
processor (all of thejson_*
functions) -
json
processor (path
field) -
process_field
processor (path
field) -
process_map
processor (premap
,premap_optional
,postmap
andpostmap_optional
fields) -
check_field
condition (path
field) -
json_field
function interpolation -
s3
input (sqs_body_path
,sqs_bucket_path
andsqs_envelope_path
fields) -
dynamodb
output (json_map_columns
field values)
Migration guide
In order to replicate the exact same behavior as currently exists your paths should be updated to include the character *
wherever an array exists. For example, the default value of sqs_body_path
for the s3
input has been updated from Records.s3.object.key
to Records.*.s3.object.key
.
Go API
Modules
Redpanda Connect now fully adheres to Go Modules, import paths must therefore now contain the major version (v3) like so:
import "github.com/Jeffail/benthos/v3/lib/processor"
It should be pretty quick to update your imports, either using a tool or just:
grep "Jeffail/benthos" . -Rl | grep -e "\.go$" | xargs -I{} sed -i 's/Jeffail\/benthos/Jeffail\/benthos\/v3/g' {}