2020-11-29 - Inter-Lambda process flows
Communication between services
Overview
When a process flow spans multiple Lambda functions and queues there are different flow models, some will benefit from a standardized model of reducing Lambda invocations using SNS queue subscription filters
Different flow models
These flows assume the sending Lambda is placing a message into the receiving Lambda service's In queue, because this implies the sending Lambda know who the receiver is and thus can expect a response. We use the Out queue when we have finished a task and do not know who might want to subscribe to be notified.
When a sending Lambda (LambdaA) requires another Lambda (LambdaB) to do processing and the flow requires additional work to be done afterwards, we do not want to invoke syncronous so will send a message to the LambdaB service's In queue to trigger, but then need to trigger a third Lambda (LambdaC) within LambdaA's service to complete additional work.
So basically LambdaA triggers LambdaB which triggers LambdaC. LambdaA and LambdaC do not have to be in the same service but in the context of flow type 3 below they often are.
1) Send and forget
If the sending Lambda does not need to receive a response it can invoke the receiving Lambda without any need to subscribe for the result.
2) Part of a contained flow process
Often LambdaB will only ever do work as part of a single multi-Lambda process, LambdaB is only ever triggered from one Lambda (LambdaA) and never from others, and its response only ever triggers LambdaC. In this case we can set up a simple subscription to LambdaB'a Out queue that triggers LambdaC, or have LambdaB add a message to LambdaA/LambdaC's service's In queue which triggers LambdaC.
3) Multiple Lambdas invoke LambdaB and only want to receive their responses
Often the work done by LambdaB is not contained to a single process, any number of Lambda's from any number of services could invoke it, normally in this situation those services only want to continue processing LambdaB results that originated from their own Lambdas.
(? not any more because of lambda filter policy limits) If we have every service that needs to process LambdaB's result use a simple subscription they would all receive and need to check the results originating from every service including themselves. To solve this and have each service's subscription only trigger when a request originated from that service we apply a requestService correlation id.