Service - Activity Switchboard: Difference between revisions
Jump to navigation
Jump to search
(→Ideas) |
No edit summary |
||
(19 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
= Overview = | = Overview = | ||
Receives most messages sent within the entire project's serverless flow, allows services to register triggers that | Receives most messages sent within the entire project's serverless flow, allows services to register a set of triggers that are checked when each message is received, if all triggers pass a message is sent out for subscribed services. | ||
= Repository = | = Repository = | ||
Line 9: | Line 9: | ||
= DynamoDB tables = | = DynamoDB tables = | ||
== | == TriggerGroups == | ||
; triggerGroupId (partition key) | |||
: string | |||
: comes from: {receiverTag}_{uniqueId} | |||
; receiverTag | |||
: set by the creating service name eg. NotificationMgr | |||
; uniqueId | |||
: comes from receiver service, eg: {notificationId}, cannot include underscores | |||
; triggers | |||
: array of objects (DynamoDB list?) | |||
: each element has triggerId, triggerGroupId, valueType, propertyName, propertyValueString or propertyValueNumber | |||
== Triggers == | |||
== | ; triggerId (partition key) | ||
: string | |||
: comes from: {hash of valueType(property|attribute).propertyName}_{hash of propertyValue} | |||
: or: serviceName_{hash of serviceName value} | |||
: or: topicName_{hash of topicName value} | |||
; triggerGroupId | |||
: string | |||
: create by [[#triggerGroup]] | |||
; valueType | |||
: string | |||
: property|attribute|serviceName|topicName | |||
; propertyName | |||
: "serviceName" and "topicName" valueTypes do not have propertyName | |||
: eg. property.propA.propB.propC | |||
: eg. attribute.serviceName | |||
; propertyValueString | |||
: type of this propertyValueString is string from valueString in [[Service - Notification Manager]] | |||
; propertyValueNumber | |||
: type of this propertyValueNumber is number from valueNumber in [[Service - Notification Manager]] | |||
= Object Schemas = | |||
; Additional Information: [[Per Service Schemas]] | |||
== objType == | |||
=== triggerGroup === | |||
* Groups many triggers, all triggers for a trigger group must pass for the trigger group to pass | * Groups many triggers, all triggers for a trigger group must pass for the trigger group to pass | ||
* no sort key | * no sort key | ||
* if multiple values are sent for a trigger we consider this an "or" check, any of the values can match. We do this by saving each value as a separate trigger, then when checking a trigger groups child triggers we have a counter that enables a check if any one of the values matches | |||
<syntaxhighlight lang="JavaScript"> | |||
{ | |||
objectType: "triggerGroup", | |||
canDelete: true, | |||
complexFilterServiceTag: "complexFilter", | |||
overWriteHandlers: { // optional, if not set will create default handlers, if empty will not create handler and main function | |||
create: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs'] | |||
update: [], // default: ['hdrApi', 'hdrSqs'] | |||
get: [], // default: ['hdrApi', 'hdrInv'] | |||
delete: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs'] | |||
}, | |||
// overwriteGeneratedMainFunction: ["create"], | |||
storageResources: { | |||
dynamoDB: { | |||
storageType: "dynamoDB", | |||
tableName: "TriggerGroups" | |||
} | |||
}, | |||
fieldNames: { // see Per Service Schemas | |||
triggers: { | |||
type: "array", | |||
requiredOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
}, | |||
receiverTag: { | |||
type: "string", | |||
requiredOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
}, | |||
uniqueId: { | |||
type: "string", | |||
requiredOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
} | |||
}, | |||
identifiers: [ | |||
{ | |||
type: "partitionKey", | |||
fieldNames: ["receiverTag", "uniqueId"], | |||
name: "triggerGroupId" | |||
} | |||
] | |||
} | |||
</syntaxhighlight> | |||
=== | ==== fieldNames ==== | ||
; triggerGroupId | ; triggerGroupId | ||
: (partition key) | : (partition key) | ||
: comes from: {receiverTag}_{uniqueId} | : comes from: {receiverTag}_{uniqueId} | ||
: uniqueId comes from receiver service, eg: {notificationId}, cannot include underscores | ; receiverTag | ||
; | : set by the creating service name eg. NotificationMgr | ||
; uniqueId | |||
: comes from receiver service, eg: {notificationId}, cannot include underscores | |||
; triggers | |||
: array of objects (DynamoDB list?) | : array of objects (DynamoDB list?) | ||
: each element has triggerId, propertyName, and | : each element has triggerId, triggerGroupId, valueType, propertyName, propertyValueString and propertyValueNumber | ||
=== trigger === | |||
= | <syntaxhighlight lang="JavaScript"> | ||
{ | |||
objectType: "trigger", | |||
canDelete: true, | |||
complexFilterServiceTag: "complexFilter", | |||
overWriteHandlers: { // optional, if not set will create default handlers, if empty will not create handler and main function | |||
create: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs'] | |||
update: [], // default: ['hdrApi', 'hdrSqs'] | |||
get: [], // default: ['hdrApi', 'hdrInv'] | |||
delete: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs'] | |||
}, | |||
// overwriteGeneratedMainFunction: ["create", "delete"], | |||
storageResources: { | |||
dynamoDB: { | |||
storageType: "dynamoDB", | |||
tableName: "Triggers" | |||
} | |||
}, | |||
fieldNames: { // see Per Service Schemas | |||
triggerId: { | |||
type: "string", | |||
optionalOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
}, | |||
triggerGroupId: { | |||
type: "string", | |||
requiredOnCreate: true, | |||
canUpdate: false, | |||
storageResourceTags: ['dynamoDB'], | |||
fromServiceNameTag: "ActivitySwitchboard", | |||
fromObjectType: "triggerGroup" | |||
}, | |||
valueType: { | |||
type: "string", | |||
requiredOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
}, | |||
propertyValueString: { | |||
type: "string", | |||
optionalOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
}, | |||
propertyValueNumber: { | |||
type: "number", | |||
optionalOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
}, | |||
propertyName: { | |||
type: "string", | |||
optionalOnCreate: true, | |||
canUpdate: false, | |||
validation: { | |||
pattern: pattern | |||
}, | |||
storageResourceTags: ['dynamoDB'] | |||
} | |||
}, | |||
identifiers: [ | |||
{ | |||
type: "partitionKey", | |||
fieldName: "triggerId" | |||
}, | |||
{ | |||
type: "sortKey", | |||
fieldName: "triggerGroupId" | |||
} | |||
] | |||
} | |||
</syntaxhighlight> | |||
=== | ==== fieldNames ==== | ||
; triggerId | ; triggerId | ||
: (partition key) | : (partition key) | ||
: comes from: { | : comes from: {hash of valueType(property|attribute).propertyName}_{hash of propertyValue} | ||
: attributes is for message attributes | : or: serviceName_{hash of serviceName value} | ||
: or: topicName_{hash of topicName value} | |||
: attributes is for message attributes | |||
: property is from the data sent inside the message body | : property is from the data sent inside the message body | ||
: hash property name and value so no ambiguity about underscores/spaces etc.. | : hash property name and value so no ambiguity about underscores/spaces etc.. | ||
; triggerGroupId | ; triggerGroupId | ||
: (sort key) | : (sort key) | ||
: created by [[#triggerGroup]] | |||
; valueType | |||
: property|attribute|serviceName|topicName | |||
; propertyName | ; propertyName | ||
: "serviceName" and "topicName" valueTypes do not have propertyName | |||
: eg. property.propA.propB.propC | |||
: eg. attribute.serviceName | |||
; propertyValueString | |||
: type of this propertyValueString is string from valueString in [[Service - Notification Manager]] | |||
; propertyValueNumber | |||
: type of this propertyValueNumber is number from valueNumber in [[Service - Notification Manager]] | |||
= Efficiency = | |||
* Service will result in a large number of queries to Triggers table, every message will need to make a query for every field set as an activityTrigger | * Service will result in a large number of queries to Triggers table, every message will need to make a query for every field set as an activityTrigger | ||
* Try to make this as efficient as possible | * Try to make this as efficient as possible | ||
* To reduce number of queries made to Tiggers table we use the | * To reduce number of queries made to Tiggers table we use the msgCfg for any message received | ||
** MsgCfg sets which properties can be used as triggers, others are not queried | ** MsgCfg sets which properties can be used as triggers, others are not queried | ||
= Handling msgCfgs = | |||
* msgCfgs get updated from [[Service - Message Config Manager|Message Config Manager]] service, we do this by subscribing to Message Config Manager's OutMsgCfgUpdate topic | |||
* Only want to process messages from certain In/Out topics, it will be most topics but does not have to be all | |||
* Set which topics to subscribe to by setting activitySwitchboard = true in msgCfg | |||
= Ideas = | = Ideas = | ||
Line 68: | Line 254: | ||
* If more efficient can use cache for regular DynamoDB queries | * If more efficient can use cache for regular DynamoDB queries | ||
* Could add other matching methods such as greater than or less than, in DynamoDB this might be more efficient to add as separate table with its own structure, eg: partition key is the field reference, sort key is the amount, then use sort key to return matching triggers. Danger of bad partitioning in DynamoDB (or hitting limits) due to large numbers of sort keys associated with one partition key | * Could add other matching methods such as greater than or less than, in DynamoDB this might be more efficient to add as separate table with its own structure, eg: partition key is the field reference, sort key is the amount, then use sort key to return matching triggers. Danger of bad partitioning in DynamoDB (or hitting limits) due to large numbers of sort keys associated with one partition key | ||
* could consider how to incorporate includes or checking within a set of options set in the trigger, again might need specialized table structure | |||
* Might be able to optimise by using SNS > stream instead of SNS > SQS for all incomming messages | * Might be able to optimise by using SNS > stream instead of SNS > SQS for all incomming messages | ||
* One system level receiving service could be specialized logs, eg service per user/per product/etc | * One system level receiving service could be specialized logs, eg logs per service per user/per product/etc | ||
* | * Topic name is not fixed part of the TriggersGroup structure, allow for trigger groups that are not connected to specific topics, but can pass messages from any topic that matches other triggers | ||
= Working documents = | = Working documents = |
Latest revision as of 05:41, 13 August 2024
Overview
Receives most messages sent within the entire project's serverless flow, allows services to register a set of triggers that are checked when each message is received, if all triggers pass a message is sent out for subscribed services.
Repository
https://bitbucket.org/stb_working/activity-switchboard/src/master/
DynamoDB tables
TriggerGroups
- triggerGroupId (partition key)
- string
- comes from: {receiverTag}_{uniqueId}
- receiverTag
- set by the creating service name eg. NotificationMgr
- uniqueId
- comes from receiver service, eg: {notificationId}, cannot include underscores
- triggers
- array of objects (DynamoDB list?)
- each element has triggerId, triggerGroupId, valueType, propertyName, propertyValueString or propertyValueNumber
Triggers
- triggerId (partition key)
- string
- comes from: {hash of valueType(property|attribute).propertyName}_{hash of propertyValue}
- or: serviceName_{hash of serviceName value}
- or: topicName_{hash of topicName value}
- triggerGroupId
- string
- create by #triggerGroup
- valueType
- string
- property|attribute|serviceName|topicName
- propertyName
- "serviceName" and "topicName" valueTypes do not have propertyName
- eg. property.propA.propB.propC
- eg. attribute.serviceName
- propertyValueString
- type of this propertyValueString is string from valueString in Service - Notification Manager
- propertyValueNumber
- type of this propertyValueNumber is number from valueNumber in Service - Notification Manager
Object Schemas
- Additional Information
- Per Service Schemas
objType
triggerGroup
- Groups many triggers, all triggers for a trigger group must pass for the trigger group to pass
- no sort key
- if multiple values are sent for a trigger we consider this an "or" check, any of the values can match. We do this by saving each value as a separate trigger, then when checking a trigger groups child triggers we have a counter that enables a check if any one of the values matches
{
objectType: "triggerGroup",
canDelete: true,
complexFilterServiceTag: "complexFilter",
overWriteHandlers: { // optional, if not set will create default handlers, if empty will not create handler and main function
create: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs']
update: [], // default: ['hdrApi', 'hdrSqs']
get: [], // default: ['hdrApi', 'hdrInv']
delete: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs']
},
// overwriteGeneratedMainFunction: ["create"],
storageResources: {
dynamoDB: {
storageType: "dynamoDB",
tableName: "TriggerGroups"
}
},
fieldNames: { // see Per Service Schemas
triggers: {
type: "array",
requiredOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
},
receiverTag: {
type: "string",
requiredOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
},
uniqueId: {
type: "string",
requiredOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
}
},
identifiers: [
{
type: "partitionKey",
fieldNames: ["receiverTag", "uniqueId"],
name: "triggerGroupId"
}
]
}
fieldNames
- triggerGroupId
- (partition key)
- comes from: {receiverTag}_{uniqueId}
- receiverTag
- set by the creating service name eg. NotificationMgr
- uniqueId
- comes from receiver service, eg: {notificationId}, cannot include underscores
- triggers
- array of objects (DynamoDB list?)
- each element has triggerId, triggerGroupId, valueType, propertyName, propertyValueString and propertyValueNumber
trigger
{
objectType: "trigger",
canDelete: true,
complexFilterServiceTag: "complexFilter",
overWriteHandlers: { // optional, if not set will create default handlers, if empty will not create handler and main function
create: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs']
update: [], // default: ['hdrApi', 'hdrSqs']
get: [], // default: ['hdrApi', 'hdrInv']
delete: ['hdrSqs'], // default: ['hdrApi', 'hdrSqs']
},
// overwriteGeneratedMainFunction: ["create", "delete"],
storageResources: {
dynamoDB: {
storageType: "dynamoDB",
tableName: "Triggers"
}
},
fieldNames: { // see Per Service Schemas
triggerId: {
type: "string",
optionalOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
},
triggerGroupId: {
type: "string",
requiredOnCreate: true,
canUpdate: false,
storageResourceTags: ['dynamoDB'],
fromServiceNameTag: "ActivitySwitchboard",
fromObjectType: "triggerGroup"
},
valueType: {
type: "string",
requiredOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
},
propertyValueString: {
type: "string",
optionalOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
},
propertyValueNumber: {
type: "number",
optionalOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
},
propertyName: {
type: "string",
optionalOnCreate: true,
canUpdate: false,
validation: {
pattern: pattern
},
storageResourceTags: ['dynamoDB']
}
},
identifiers: [
{
type: "partitionKey",
fieldName: "triggerId"
},
{
type: "sortKey",
fieldName: "triggerGroupId"
}
]
}
fieldNames
- triggerId
- (partition key)
- comes from: {hash of valueType(property|attribute).propertyName}_{hash of propertyValue}
- or: serviceName_{hash of serviceName value}
- or: topicName_{hash of topicName value}
- attributes is for message attributes
- property is from the data sent inside the message body
- hash property name and value so no ambiguity about underscores/spaces etc..
- triggerGroupId
- (sort key)
- created by #triggerGroup
- valueType
- property|attribute|serviceName|topicName
- propertyName
- "serviceName" and "topicName" valueTypes do not have propertyName
- eg. property.propA.propB.propC
- eg. attribute.serviceName
- propertyValueString
- type of this propertyValueString is string from valueString in Service - Notification Manager
- propertyValueNumber
- type of this propertyValueNumber is number from valueNumber in Service - Notification Manager
Efficiency
- Service will result in a large number of queries to Triggers table, every message will need to make a query for every field set as an activityTrigger
- Try to make this as efficient as possible
- To reduce number of queries made to Tiggers table we use the msgCfg for any message received
- MsgCfg sets which properties can be used as triggers, others are not queried
Handling msgCfgs
- msgCfgs get updated from Message Config Manager service, we do this by subscribing to Message Config Manager's OutMsgCfgUpdate topic
- Only want to process messages from certain In/Out topics, it will be most topics but does not have to be all
- Set which topics to subscribe to by setting activitySwitchboard = true in msgCfg
Ideas
- If more efficient can use cache for regular DynamoDB queries
- Could add other matching methods such as greater than or less than, in DynamoDB this might be more efficient to add as separate table with its own structure, eg: partition key is the field reference, sort key is the amount, then use sort key to return matching triggers. Danger of bad partitioning in DynamoDB (or hitting limits) due to large numbers of sort keys associated with one partition key
- could consider how to incorporate includes or checking within a set of options set in the trigger, again might need specialized table structure
- Might be able to optimise by using SNS > stream instead of SNS > SQS for all incomming messages
- One system level receiving service could be specialized logs, eg logs per service per user/per product/etc
- Topic name is not fixed part of the TriggersGroup structure, allow for trigger groups that are not connected to specific topics, but can pass messages from any topic that matches other triggers