Between backend Lambda functions

Idea - SNS > SQS

SNS+SQS queues are interesting because they allow for strong retry/DLQ management
Allows for other processes to hook in, eg debug or notifications
Not sure how expensive they will be
More difficult to manage, extra services created
Good for major steps in workflow between services
SQS has good retry structure if receiving service fails
SQS can easily stop the receiving service if need to fix something, and requests queue up
SNS can add in other subscribers such as auditing/logging/notifications

Incoming API requests deliver to a lambda function that directs to the actual function.
Internal invoking can pass this function as well, allowing for standardized handling of the requests
This would allow a single access point to all actions, so eg direct invocations do not need to hardwire each function they connect to, instead they all invoke the one function with a parameter of the action needed, that one function does the redirection, so if we change where those requests go we change in one place and all dependant code will redirect
Would want execution of final logic to be async so no double charging of execution time

This is what we are currently using, can be set to lambda invoke/api/queue/.. in the Config DB table endpoint setting

Each calling method delivers a different form of the event, eg SQS delivers as an array of messages
Invoked lambda (handler) must parse the received event differently depending on the calling method
Can abstract the connection to external sources, saving the service type and connection details in a config variable, so easy to change, the function automatically handles any type of endpoint, tried this and was complicated, for example if expecting a queue the logic must process in batches, if never expecting a queue (eg expects to only be called by lambda invoke) then don’t need to code for batches
eg: Lambdas that invoke from edge API Gateway requests will only ever receive a single request, no batches, so not need to code for batch work (unless we build the frontend to send batch requests but that is a special case that can have special parsing to handle)
solution: If we have a lambda that might be called in different ways, try to keep the triggered lambda (handler) as small as possible, parsing the request event and then invoking another function to do the actual work. If is batch processing, the looping of the array is handled in the triggered handler lambda, invoking the processing function for each message
This way we could have separate triggered lambda’s for each calling method (or split them into types, eg API+Lambda invoke together, methods that work as batch together), but logic is handled in one location

On AWS we can use IAM to allow any resource access to any other
If using API Gateway can set authorization there using IAM, or CustomAuthorizer if need RBAC etc..

Try to make everything async event process that fits nicely into Step Functions
So no Step waits for a response, further processing is handled by later functions
This way no double charged compute time while a process waits for another process
Will need some way to be notified (or request update) of state of processing (for actions that the client request wants the result of the processing). This could be done by using SNS+SQS queues to pass messages between each stage, debug/notifications processes can be added to the SNS subscribers
Do not need to use AWS Step Functions, that look like it might be costly and have limits, we can probably manage as deployed functions via Serverless