2021-04-17 - Change from MsgIn MsgOut queues

Reason for changing / No longer used

Due to AWS SNS limits, specifically account level filter policy limits, the combined MsgIn/Out queue idea will not be feasible, the alternative will be a specific queue for each message type (ie: msgTag), the msgTag will no longer be needed, instead the SNS queue can be checked to see type of message by services/functions that receive multiple message types.

AWS accounts allow 200 filter policies per account, used to be 100, appears to be a relatively hard limit, not a lot of chatter about requesting increases.

• AWS account can have up to 100,000 SNS topics
• AWS account can have maximum 200 subscription policies, this makes shared MsgIn/Out queues unusable

moved from Communication between services page

MsgIn and MsgOut queues per service

• Each Amazon account has a limit of 100 SNS queues per account, by using two standardized queues per service can keep this number down
• Having one standard queue to look at for messages will make it easier/cleaner when other services need to connect to the service
• reference: https://www.jeremydaly.com/how-to-use-sns-and-sqs-to-distribute-and-throttle-events/

When to use MsgIn or MsgOut

• Sometimes we will have a parent-child relationship between services, the parent service is deployed first, and other child services that depend on it are deployed later, in this situation the child services might use the parents MsgOut SNS queue to receive messages from the parent service, and the parents MsgIn SNS queue to send messages to the parent
• Use MsgIn when expect a large number of services will be sending requests to one Lambda, eg ComplexFilter'ing
• Use MsgOut when a large number of services will be waiting for response of one Lambda (often filtered to only receive responses that originated from the recieving Lambda)
• often they are paired in a flow, the request in goes to MsgIn, the output goes to MsgOut
• Also consider whether which service it is more logical to know the endpoint, if the sending service is more logical then it will send to the receiving service's MsgIn. If receiving service is more logical it subscribtes to sending service's MsgOut
• Another way to decide is to consider whether the message is a notification of some work finishing (place in own service's MsgOut queue), or is a task being sent to a specific place to continue processing (place in other service's MsgIn queue)

Deploying/changing queue subscriptions

• When recieving service subscribes a local SQS to an external MsgOut SNS is a bit tricky, we can hardcode subscribing in intial setup for now
• Config table will have a method of recording and changing subscriptions
• When service sends messages to other service's MsgIn queue I believe can use topic name, probably built from a record in sending service's Config table. If cannot refer to and send message to SNS topic by name, we would need to store the arn of the parent service’s SNS in the child service’s Config DB.

Standardize message attributes

• Want to create a standardized structure for message attributes, which can then be used for filtering SNS topic subscriptions:

1. (? fromService - name of the service sending the message) (might not be needed)
2. msgTag - basic and unique description of the message
3. requestService - present when service names have been added to the requestServices array, the last added entry is always added as requestService to outgoing messages until the flow returns to that service, then that element is popped off the array and the next service, if any, is used. NPM module - izara-middleware#requestServices

• can then add use case specific message attributes so clients can subscribe to only receive message for the specific filter/s they are waiting for

Serverless framework deploying queues

• First time deploy will get an error on SNS subscription if it has filters, need to comment them, then can do a second deploy with them added