Cloud
Also in Self-Managed

sftp

Consumes files from an SFTP server.

Metadata

This input adds the following metadata fields to each message:

  • sftp_path

  • sftp_mod_time

You can access these metadata fields using function interpolation.

  • Common

  • Advanced

input:
  label: ""
  sftp:
    address: "" # No default (required)
    credentials:
      username: ""
      password: ""
      host_public_key_file: "" # No default (optional)
      host_public_key: "" # No default (optional)
      private_key_file: "" # No default (optional)
      private_key: "" # No default (optional)
      private_key_pass: ""
    paths: [] # No default (required)
    auto_replay_nacks: true
    scanner:
      to_the_end: {}
    watcher:
      enabled: false
      minimum_age: 1s
      poll_interval: 1s
      cache: ""
input:
  label: ""
  sftp:
    address: "" # No default (required)
    connection_timeout: 30s
    credentials:
      username: ""
      password: ""
      host_public_key_file: "" # No default (optional)
      host_public_key: "" # No default (optional)
      private_key_file: "" # No default (optional)
      private_key: "" # No default (optional)
      private_key_pass: ""
      ssh_algorithms:
        additional_key_exchanges: [] # No default (optional)
        additional_ciphers: [] # No default (optional)
        additional_macs: [] # No default (optional)
    max_sftp_sessions: 10
    paths: [] # No default (required)
    auto_replay_nacks: true
    scanner:
      to_the_end: {}
    delete_on_finish: false
    watcher:
      enabled: false
      minimum_age: 1s
      poll_interval: 1s
      cache: ""
      cache_mode: path

Fields

address

The address (hostname or IP address) of the SFTP server to connect to.

Type: string

auto_replay_nacks

Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure.

Set auto_replay_nacks to false to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.

Type: bool

Default: true

connection_timeout

The connection timeout to use when connecting to the target server.

Type: string

Default: 30s

credentials

The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access.

Type: object

credentials.host_public_key

The raw contents of the SFTP server’s public key, used for host key verification.

Type: string

credentials.host_public_key_file

The path to the SFTP server’s public key file, used for host key verification.

Type: string

credentials.password

The password to use for authentication. Used together with username for basic authentication or with encrypted private keys for secure access.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

credentials.private_key

The raw contents of the private key used to authenticate with the SFTP server. This field provides an alternative to private_key_file.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

credentials.private_key_file

The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the private_key field.

Type: string

credentials.private_key_pass

An optional passphrase for decrypting the private key, if it is encrypted.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

credentials.ssh_algorithms

Additional SSH transport algorithms to permit for compatibility with legacy SFTP servers. Configured algorithms are appended to the default algorithms, which remain preferred, and when unset the defaults are used unchanged. Algorithms the underlying SSH library classifies as insecure weaken transport security and should only be enabled when required by a known server. These settings do not affect host key verification.

Type: object

credentials.ssh_algorithms.additional_ciphers[]

Ciphers to permit in addition to the defaults, for example aes128-cbc.

Type: array<string>

credentials.ssh_algorithms.additional_key_exchanges[]

Key exchange algorithms to permit in addition to the defaults, for example diffie-hellman-group1-sha1 or diffie-hellman-group-exchange-sha1.

Type: array<string>

credentials.ssh_algorithms.additional_macs[]

MAC algorithms to permit in addition to the defaults.

Type: array<string>

credentials.username

The username required to authenticate with the SFTP server.

Type: string

Default: ""

delete_on_finish

Whether to delete files from the server once they are processed.

Type: bool

Default: false

max_sftp_sessions

The maximum number of SFTP sessions.

Type: int

Default: 10

paths[]

A list of paths to consume sequentially. Glob patterns are supported.

Type: array<string>

scanner

The scanner used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the csv scanner processes individual rows in a CSV file without loading the entire file in memory.

Type: scanner

Default:

to_the_end: {}

watcher

An experimental mode whereby the input will periodically scan the target paths for new files and consume them, when all files are consumed the input will continue polling for new files.

Type: object

watcher.cache

A cache resource for storing the paths of files already consumed.

Type: string

Default: ""

watcher.cache_mode

Controls what the watcher stores in the cache for each consumed file. path records only the path, so a consumed file is never consumed again. path_and_mod_time also records the file’s modification time, and consumes a previously consumed file again if its modification time changes. Re-consumed files are emitted again in full, so downstream consumers may see duplicate data; the sftp_mod_time metadata field can be used to tell versions apart. SFTP reports modification times in whole seconds, so changes are only reliably detected when minimum_age is at least 1s. When an existing cache is switched to path_and_mod_time, each entry written in path mode is overwritten in the cache with the file’s current modification time the next time the file is polled. Only the cache entry changes: the file is not read again, nothing is sent to the output, and the file on the SFTP server is not modified. With a topic-backed cache such as redpanda, this produces one new record per file to the cache topic, not to the pipeline’s output. This cannot be undone, so a file that was modified after it was last consumed but before the switch is never consumed again unless its cache entry is removed. Downgrading to a Redpanda Connect version without cache_mode while files are pending leaves those files unconsumed until their cache entries are removed.

Type: string

Default: path

Options: path, path_and_mod_time

watcher.enabled

Whether file watching is enabled.

Type: bool

Default: false

watcher.minimum_age

The minimum period of time since a file was last updated before attempting to consume it. Increasing this period decreases the likelihood that a file will be consumed whilst it is still being written to.

Type: string

Default: 1s

# Examples:
minimum_age: 10s

# ---

minimum_age: 1m

# ---

minimum_age: 10m

watcher.poll_interval

The interval between each attempt to scan the target paths for new files.

Type: string

Default: 1s

# Examples:
poll_interval: 100ms

# ---

poll_interval: 1s