aws_lambda
Invokes an AWS lambda for each message. The contents of the message is the payload of the request, and the result of the invocation will become the new contents of the message.
Introduced in version 3.36.0.
The rate_limit field can be used to specify a rate limit resource to cap the rate of requests across parallel components service wide.
In order to map or encode the payload to a specific request body, and map the response back into the original payload instead of replacing it entirely, you can use the branch processor.
Error handling
When Redpanda Connect is unable to connect to the AWS endpoint or is otherwise unable to invoke the target lambda function it will retry the request according to the configured number of retries. Once these attempts have been exhausted the failed message will continue through the pipeline with it’s contents unchanged, but flagged as having failed, allowing you to use standard processor error handling patterns.
However, if the invocation of the function is successful but the function itself throws an error, then the message will have it’s contents updated with a JSON payload describing the reason for the failure, and a metadata field lambda_function_error will be added to the message allowing you to detect and handle function errors with a branch:
pipeline:
processors:
- branch:
processors:
- aws_lambda:
function: foo
result_map: |
root = if meta().exists("lambda_function_error") {
throw("Invocation failed due to %v: %v".format(this.errorType, this.errorMessage))
} else {
this
}
output:
switch:
retry_until_success: false
cases:
- check: errored()
output:
reject: ${! error() }
- output:
resource: somewhere_elseCredentials
By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in Amazon Web Services.
-
Common
-
Advanced
processor:
label: ""
aws_lambda:
parallel: false
function: "" # No default (required)processor:
label: ""
aws_lambda:
parallel: false
function: "" # No default (required)
rate_limit: ""
region: "" # No default (optional)
endpoint: "" # No default (optional)
tcp:
connect_timeout: 0s
keep_alive:
idle: 15s
interval: 15s
count: 9
tcp_user_timeout: 0s
credentials:
profile: "" # No default (optional)
id: "" # No default (optional)
secret: "" # No default (optional)
token: "" # No default (optional)
from_ec2_role: false # No default (optional)
role: "" # No default (optional)
role_external_id: "" # No default (optional)
timeout: 5s
retries: 3Examples
Branched Invoke
This example uses a branch processor to map a new payload for triggering a lambda function with an ID and username from the original message, and the result of the lambda is discarded, meaning the original message is unchanged.
pipeline:
processors:
- branch:
request_map: '{"id":this.doc.id,"username":this.user.name}'
processors:
- aws_lambda:
function: trigger_user_updateFields
credentials
Manually configure the AWS credentials to use (optional). For more information, see the Amazon Web Services guide.
Type: object
credentials.from_ec2_role
Use the credentials of a host EC2 machine configured to assume an IAM role associated with the instance.
Requires version 4.2.0 or later.
Type: bool
credentials.secret
The secret for the AWS credentials in use.
|
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets. |
Type: string
credentials.token
The token for the AWS credentials in use. Required only when using short-term credentials.
Type: string
endpoint
A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints.
Type: string
tcp
Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for:
-
Unresponsive hosts: Set
connect_timeoutto limit how long a connection attempt can take (the default0ssets no limit) -
Long-lived connections: Configure
keep_alivesettings to detect and recover from stale connections -
Unstable networks: Tune keep-alive probes to balance between quick failure detection and avoiding false positives
-
Linux systems with specific requirements: Use
tcp_user_timeout(Linux 2.6.37+) to control data acknowledgment timeouts
Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.
Requires version 4.69.0 or later.
Type: object
tcp.connect_timeout
Maximum amount of time a dial will wait for a connect to complete. Zero disables.
Type: string
Default: 0s
tcp.keep_alive.count
Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9.
Type: int
Default: 9
tcp.keep_alive.idle
Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes.
Type: string
Default: 15s
tcp.keep_alive.interval
Duration between keep-alive probes. Zero defaults to 15s.
Type: string
Default: 15s
