Service - Activity Switchboard

Overview

Receives most messages sent within the entire project's serverless flow, allows services to register a set of triggers that are checked when each message is received, if all triggers pass a message is sent out for subscribed services.

Repository

https://bitbucket.org/stb_working/activity-switchboard/src/master/

DynamoDB tables

Standard Config Table Per Service

Configuration tags

{
	configKey: "MsgCfg"
	configTag: {serviceName}_{topicName}
	configValue: {msgCfg}
}

propogated from Message Config Manager service

see Standard message config for In Out topics

TriggerGroups

• Groups many triggers, all triggers for a trigger group must pass for the trigger group to pass
• no sort key

Fields

triggerGroupId

(partition key)

comes from: {receiverTag}_{uniqueId}

uniqueId comes from receiver service, eg: {notificationId}, cannot include underscores

triggers

array of objects (DynamoDB list?)

each element has triggerId, propertyName, and propertyValue

additionalData

set by the receiving service, gets added to activity messages send to receiving service

could include id/s needed by the receiving service to match the trigger to its entity

Triggers

Fields

triggerId

(partition key)

comes from: {"attributes"|"property"}_{hash of propertyName}_{hash of propertyValue}

attributes is for message attributes

property is from the data sent inside the message body

hash property name and value so no ambiguity about underscores/spaces etc..

triggerGroupId

(sort key)

propertyName

propertyValue

Efficiency

• Service will result in a large number of queries to Triggers table, every message will need to make a query for every field set as an activityTrigger
• Try to make this as efficient as possible
• To reduce number of queries made to Tiggers table we use the msgCfg for any message received
  • MsgCfg sets which properties can be used as triggers, others are not queried

Handling msgCfgs

• msgCfgs get updated from Message Config Manager service, we do this by subscribing to Message Config Manager's OutMsgCfgUpdate topic
• Only want to process messages from certain In/Out topics, it will be most topics but does not have to be all
• Set which topics to subscribe to by creating a "MsgCfg" record in the Config table, initially the configValue will be empty and needs to be pulled from Message Config Manager
• Plan is to set a list of topics in InitalSetup, for each add a record to Config table and invoking 2020-11-08 - Message Config Manager - Functions#getMsgCfgs or getMsgCfg, not sure how to do this to ensure the configs already exist
• Could also perhaps have a placeholder configValue that states the MsgCfg has not yet been dug, when a message comes in if that is set, request the MsgCfg then, if the request is unsuccessful have another placeholder that has timestamp when last checked, after a certain time re-request (and probably DLQ the message that arrived)

Ideas

• If more efficient can use cache for regular DynamoDB queries
• Could add other matching methods such as greater than or less than, in DynamoDB this might be more efficient to add as separate table with its own structure, eg: partition key is the field reference, sort key is the amount, then use sort key to return matching triggers. Danger of bad partitioning in DynamoDB (or hitting limits) due to large numbers of sort keys associated with one partition key
• could consider how to incorporate includes or checking within a set of options set in the trigger, again might need specialized table structure
• Might be able to optimise by using SNS > stream instead of SNS > SQS for all incomming messages
• One system level receiving service could be specialized logs, eg logs per service per user/per product/etc
• Topic name is not fixed part of the TriggersGroup structure, allow for trigger groups that are not connected to specific topics, but can pass messages from any topic that matches other triggers

Working documents

Working_documents - Activity Switchboard