Communication between services
Revision as of 07:51, 4 December 2020 by Sven the Barbarian (talk | contribs) (→Standardize message attributes)
Overview
How to pass tasks between services
Communication between backend services
MsgIn and MsgOut queues per service
- Each Amazon account has a limit of 100 SNS queues per account, by using two standardized queues per service can keep this number down
- Having one standard queue to look at for messages will make it easier/cleaner when other services need to connect to the service
- reference: https://www.jeremydaly.com/how-to-use-sns-and-sqs-to-distribute-and-throttle-events/
When to use MsgIn or MsgOut
- Sometimes we will have a parent-child relationship between services, the parent service is deployed first, and other child services that depend on it are deployed later, in this situation the child services might use the parents MsgOut SNS queue to receive messages from the parent service, and the parents MsgIn SNS queue to send messages to the parent
- Use MsgIn when expect a large number of services will be sending requests to one Lambda, eg ComplexFilter'ing
- Use MsgOut when a large number of services will be waiting for response of one Lambda (often filtered to only receive responses that originated from the recieving Lambda)
- often they are paired in a flow, the request in goes to MsgIn, the output goes to MsgOut
- Also consider whether which service it is more logical to know the endpoint, if the sending service is more logical then it will send to the receiving service's MsgIn. If receiving service is more logical it subscribtes to sending service's MsgOut
- Another way to decide is to consider whether the message is a notification of some work finishing (place in own service's MsgOut queue), or is a task being sent to a specific place to continue processing (place in other service's MsgIn queue)
Deploying/changing queue subscriptions
- When recieving service subscribes a local SQS to an external MsgOut SNS is a bit tricky, we can hardcode subscribing in intial setup for now
- Config table will have a method of recording and changing subscriptions
- When service sends messages to other service's MsgIn queue I believe can use topic name, probably built from a record in sending service's Config table. If cannot refer to and send message to SNS topic by name, we would need to store the arn of the parent service’s SNS in the child service’s Config DB.
Standardize message attributes
- Want to create a standardized structure for message attributes, which can then be used for filtering SNS topic subscriptions:
- (? fromService - name of the service sending the message) (might not be needed)
- msgTag - basic and unique description of the message
- requestService - present when service names have been added to the requestServices array, the last added entry is always added as requestService to outgoing messages until the flow returns to that service, then that element is popped off the array and the next service, if any, is used. NPM module - izara-middleware#requestServices
- can then add use case specific message attributes so clients can subscribe to only receive message for the specific filter/s they are waiting for
Serverless framework deploying queues
- First time deploy will get an error on SNS subscription if it has filters, need to comment them, then can do a second deploy with them added
Direct Lambda invocations
- Not as common as MsgIn/MsgOut queue flow
- The queue flow ensures safer delivery of requests, can be paused (accumulate messages that come in), and can hook in additional services to monitor the messages
- For flows that need to be heavily optimised is a possibility
- Sometimes is useful for syncronous requests can be used, eg Authorizer that gets called on each API Gateway request that checks RBAC
Direct send to SQS queue that triggers Lambda
- If communication is part of a single unit of work and we want to be more efficient, are sure we only ever want one Lambda to receive the request, we could skip SNS queue and place directly on SQS for the Lambda
Return values from Lambda functions
These are managed in the handler Lambda function, return will be different depending on the requesting resource type:
API Handlers
- Use response function that sets up StatusCode/Headers/Body
- I believe the entire response object is placed into a “body” parameter of the response sent to client
- API Gateway adds its own status code, this is always 200 if the Lambda function was invoked (even if throws error) I think, might map status code from response now…
Direct invoke
- If invoked async response is discarded, but some handlers will receive requests both sync and async so OK to create response objects
- Response is placed into an object with a StatusCode property (200 if Lambda invoked successfully, code likely to change if Lambda fails before invoking, eg concurrency limits)
- Response is placed into Payload property, stringified, need to json.parse
- Our middleware formats the response into (err, response) before returning
- If Lambda throws error (eg middleware validation) response is placed into err
- Try to throw errors for any unexpected, eg config/validation type errors, other expected checks should return in the response
- I believe we tested Lambda return Error object is the same as throwing an error. Callback with first argument set as Error also treated as an error
- Tested return (x,y) only returns y (last variable), maybe our middleware messing with this
- If return or throw an Error object, or callback with an error as first return value, Lambda will add a top level parameter UnhandledFunctionError, Payload is the error object
SQS triggers Lambda
When using an SQS queue to trigger Lambda functions a batch of messages can be sent, if the Lambda throws an error or times out all messages in the batch are retried, otherwise all are deleted.
Our method of managing this effectively is to return normally from the Lambda function, and any messages that fail we manually re-add them to the SQS, or send to DLQ if too many retries.
If re-adding any messages to the SQS queue fails we could throw an error from the handler function, meaning all messages get retried using the queues retry configuration, this would ensure no processes are lost but would need to ensure function is idempotent (which is always should be anyway).