sftp
Consumes files from an SFTP server.
Metadata
This input adds the following metadata fields to each message:
-
sftp_path -
sftp_mod_time
You can access these metadata fields using function interpolation.
-
Common
-
Advanced
input:
label: ""
sftp:
address: "" # No default (required)
credentials:
username: ""
password: ""
host_public_key_file: "" # No default (optional)
host_public_key: "" # No default (optional)
private_key_file: "" # No default (optional)
private_key: "" # No default (optional)
private_key_pass: ""
paths: [] # No default (required)
auto_replay_nacks: true
scanner:
to_the_end: {}
watcher:
enabled: false
minimum_age: 1s
poll_interval: 1s
cache: ""input:
label: ""
sftp:
address: "" # No default (required)
connection_timeout: 30s
credentials:
username: ""
password: ""
host_public_key_file: "" # No default (optional)
host_public_key: "" # No default (optional)
private_key_file: "" # No default (optional)
private_key: "" # No default (optional)
private_key_pass: ""
ssh_algorithms:
additional_key_exchanges: [] # No default (optional)
additional_ciphers: [] # No default (optional)
additional_macs: [] # No default (optional)
max_sftp_sessions: 10
paths: [] # No default (required)
auto_replay_nacks: true
scanner:
to_the_end: {}
delete_on_finish: false
watcher:
enabled: false
minimum_age: 1s
poll_interval: 1s
cache: ""Fields
auto_replay_nacks
Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure.
Set auto_replay_nacks to false to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.
Type: bool
Default: true
connection_timeout
The connection timeout to use when connecting to the target server.
Type: string
Default: 30s
credentials
The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access.
Type: object
credentials.host_public_key
The raw contents of the SFTP server’s public key, used for host key verification.
Type: string
credentials.host_public_key_file
The path to the SFTP server’s public key file, used for host key verification.
Type: string
credentials.password
The password to use for authentication. Used together with username for basic authentication or with encrypted private keys for secure access.
|
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
credentials.private_key
The raw contents of the private key used to authenticate with the SFTP server. This field provides an alternative to private_key_file.
|
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
credentials.private_key_file
The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the private_key field.
Type: string
credentials.private_key_pass
An optional passphrase for decrypting the private key, if it is encrypted.
|
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
credentials.ssh_algorithms
Additional SSH transport algorithms to permit for compatibility with legacy SFTP servers. Configured algorithms are appended to the default algorithms, which remain preferred, and when unset the defaults are used unchanged. Algorithms the underlying SSH library classifies as insecure weaken transport security and should only be enabled when required by a known server. These settings do not affect host key verification.
Type: object
credentials.ssh_algorithms.additional_ciphers[]
Ciphers to permit in addition to the defaults, for example aes128-cbc.
Type: array<string>
credentials.ssh_algorithms.additional_key_exchanges[]
Key exchange algorithms to permit in addition to the defaults, for example diffie-hellman-group1-sha1 or diffie-hellman-group-exchange-sha1.
Type: array<string>
credentials.ssh_algorithms.additional_macs[]
MAC algorithms to permit in addition to the defaults.
Type: array<string>
credentials.username
The username required to authenticate with the SFTP server.
Type: string
Default: ""
delete_on_finish
Whether to delete files from the server once they are processed.
Type: bool
Default: false
scanner
The scanner used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the csv scanner processes individual rows in a CSV file without loading the entire file in memory.
Type: scanner
Default:
to_the_end: {}watcher
An experimental mode whereby the input will periodically scan the target paths for new files and consume them, when all files are consumed the input will continue polling for new files.
Type: object
watcher.cache
A cache resource for storing the paths of files already consumed.
Type: string
Default: ""
watcher.minimum_age
The minimum period of time since a file was last updated before attempting to consume it. Increasing this period decreases the likelihood that a file will be consumed whilst it is still being written to.
Type: string
Default: 1s
# Examples:
minimum_age: 10s
# ---
minimum_age: 1m
# ---
minimum_age: 10m