Communication between services

Overview

How to pass tasks between services

Communication between backend services

MsgIn and MsgOut queues per service

• Each Amazon account has a limit of 100 SNS queues per account, by using two standardized queues per service can keep this number down
• Having one standard queue to look at for messages will make it easier/cleaner when other services need to connect to the service
• reference: https://www.jeremydaly.com/how-to-use-sns-and-sqs-to-distribute-and-throttle-events/

When to use MsgIn or MsgOut

• Sometimes we will have a parent-child relationship between services, the parent service is deployed first, and other child services that depend on it are deployed later, in this situation the child services might use the parents MsgOut SNS queue to receive messages from the parent service, and the parents MsgIn SNS queue to send messages to the parent
• Use MsgIn when expect a large number of services will be sending requests to one Lambda, eg ComplexFilter'ing
• Use MsgOut when a large number of services will be waiting for response of one Lambda (often filtered to only receive responses that originated from the recieving Lambda)
• often they are paired in a flow, the request in goes to MsgIn, the output goes to MsgOut
• Also consider whether which service it is more logical to know the endpoint, if the sending service is more logical then it will send to the receiving service's MsgIn. If receiving service is more logical it subscribtes to sending service's MsgOut
• Another way to decide is to consider whether the message is a notification of some work finishing (place in own service's MsgOut queue), or is a task being sent to a specific place to continue processing (place in other service's MsgIn queue)

Deploying/changing queue subscriptions

• When recieving service subscribes a local SQS to an external MsgOut SNS is a bit tricky, we can hardcode subscribing in intial setup for now
• Config table will have a method of recording and changing subscriptions
• When service sends messages to other service's MsgIn queue I believe can use topic name, probably built from a record in sending service's Config table. If cannot refer to and send message to SNS topic by name, we would need to store the arn of the parent service’s SNS in the child service’s Config DB.

Standardize message attributes

• Want to create a standardized structure for message attributes, which can then be used for filtering SNS topic subscriptions
• Three standard attributes:
  • fromService - name of the service sending the message (might not be needed)
  • messageCategory - general description of the type of message, eg: filter_flow, or admin_change
  • messageType - specific description of the message, eg completed_filter, or update_config_variable
  • can then add use case specific message attributes, such as the filter_id of the complexfilter flow, so eg clients can subscribe to only receive message for the one filter they are waiting for
    • requestService - array/stack of services that are waiting for a response
      • Often we will only want a subscription to process responses that were sent by that specific service, add this attribute and the receiving function/s must pass it back out in their resulting MsgOut response message, it is then used by the requesting service to filter to only receive messages it initiated
      • Each service that passes down a message passes on this attribute if it exists and (if needed) adds its own service name
      • As the replies come back up the stack they remove their own service name from the array and continue passing up

Serverless framework deploying queues

• First time deploy will get an error on SNS subscription if it has filters, need to comment them, then can do a second deploy with them added

Direct Lambda invocations

• Not as common as MsgIn/MsgOut queue flow
• The queue flow ensures safer delivery of requests, can be paused (accumulate messages that come in), and can hook in additional services to monitor the messages
• For flows that need to be heavily optimised is a possibility
• Sometimes is useful for syncronous requests can be used, eg Authorizer that gets called on each API Gateway request that checks RBAC

Direct send to SQS queue that triggers Lambda

• If communication is part of a single unit of work and we want to be more efficient, are sure we only ever want one Lambda to receive the request, we could skip SNS queue and place directly on SQS for the Lambda

Return values from Lambda functions

These are managed in the handler Lambda function, return will be different depending on the requesting resource type:

API Handlers

• Use response function that sets up StatusCode/Headers/Body
• I believe the entire response object is placed into a “body” parameter of the response sent to client
• API Gateway adds its own status code, this is always 200 if the Lambda function was invoked (even if throws error) I think, might map status code from response now…

Direct invoke

• If invoked async response is discarded, but some handlers will receive requests both sync and async so OK to create response objects
• Response is placed into an object with a StatusCode property (200 if Lambda invoked successfully, code likely to change if Lambda fails before invoking, eg concurrency limits)
• Response is placed into Payload property, stringified, need to json.parse
• Our middleware formats the response into (err, response) before returning
• If Lambda throws error (eg middleware validation) response is placed into err
• Try to throw errors for any unexpected, eg config/validation type errors, other expected checks should return in the response
• I believe we tested Lambda return Error object is the same as throwing an error. Callback with first argument set as Error also treated as an error
• Tested return (x,y) only returns y (last variable), maybe our middleware messing with this
• If return or throw an Error object, or callback with an error as first return value, Lambda will add a top level parameter UnhandledFunctionError, Payload is the error object

SQS triggers Lambda

When using an SQS queue to trigger Lambda functions a batch of messages can be sent, if the Lambda throws an error or times out all messages in the batch are retried, otherwise all are deleted.

Our method of managing this effectively is to return normally from the Lambda function, and any messages that fail we manually re-add them to the SQS, or send to DLQ if too many retries.

If re-adding any messages to the SQS queue fails we could throw an error from the handler function, meaning all messages get retried using the queues retry configuration, this would ensure no processes are lost but would need to ensure function is idempotent (which is always should be anyway).

Working documents

Working documents - Communication between services