PlugIn Project Integration

Overview

Creating connections between projects with code hooks that send data between them.

Terminology

• Main project: Project the has the full plugIn schema and initiates connections
• Sub project: Project that has the reference plugIn schema
• Sending project: Project that has a plugInHook and sends request to another project
• Receiving project: Project that receives the request from the sending project
• Base objType: The obType set in plugIn schema, both sides of the plugIn have a base objType, the main project is the top level objType in plugIn schema, for the sub project it is the externalConfig objType

PlugIns can be bi-directional, so the main and sub projects can be both sending and receiving projects.

Structure

PlugIn Schema

• each project adds it's available plugIn schemas to S3
• plugIn is identified by a plugInTag
• a plugIn connects only one project to another project
• if multiple projects can connect to a single project seperate each as it's own plugin with it's own lambdas/flows, shared code in libs

PlugInHookTags

• plugInHookTags are defined by the code in that project
• plugIns choose which plugInHookTags that want to hook into
• for this reason we can identify plugInHooks by their plugInHookTag only, within one project each plugInHookTag must be unique
• the receiving project will also generate websocket actions/api endpoints/flows and lambdas according to the sending projects plugInHookTags
• the receiving project's generated resources also include the serviceTag which comes from the other project's plugInSchema objType
• so different plugIns that send to the same project>service>objType must be careful not to duplicate plugInHookTag names

Schema files

• both projects for the plugIn have an S3 schema file for a plugIn
• the main project has the full schema
• the sub project lists a reference to the main project's schema
• each plugIn has it's own file in the projects schema, filename same plugInTag

Example Core plugIn Schema

{
	connectionInitiatedBy: "user", // user|system
	objType:{
		serviceTag: "Business",
		objectType: "business"
	},
    inPlugInHookTags:{
		// same structure as outPlugInHookTags
    },
    outPlugInHookTags:{
		xxx: { // plugInHookTag
			invocationType: "async",
		},
		yyy: {
			invocationType: "sync",
		}
    },
	externalConfig:{
		createInstance: true, // setting of external createInstance , internal 
		objType:{
			serviceTag: "ComAcc",
			objectType: "comAccLinkByBusiness"
		},
		sendFieldNamesOnCreate:{
			dailyRateCurrencyTableId: { // name of param sent to sub project
				source: "baseObject", // comes from the main projects base objType
				fieldName: "dailyRateCurrencyTableId",
				canEdit: true // default false, whether user can adjust value when making plugin connection, if false forced same as base object
			},
			functionalCurrency: { // name of param sent to sub project
				source: "baseObject", // comes from the main projects base objType
				fieldName: "functionalCurrencyId"
			},
			precision: { // name of param sent to sub project
				source: "baseObject", // comes from the main projects base objType
				fieldName: "precision"
			},
			customField: {
				source: "custom", // similar to objType fieldNames, but only basic properties
				type: "string",
				validation: {
					pattern: pattern
				},		
			}
		},
		schemaBucketUrl: "object-schema.s3.us-east-2.amazonaws.com"
		snsArnPrefix: "arn:aws:sns:us-east-2:860226617901:",
		//      websocketUrl: "wss://kw86izcde3.execute-api.us-east-2.amazonaws.com/"
		apiBaseUrl: "https://1y9v1t7rvc.execute-api.us-east-2.amazonaws.com/" // for full endpoint add: /{serviceTag}/{plugInTag}_{plugInHookTag}
    }
}

• connectionInitiatedBy sets whether the plugIn connection flow in the main project is initiatedBy user or system, eg commerce>comAcc is user, comAcc>accounting is system

Example Reference plugIn Schema

{
   "schemaBucketUrl": "object-schema.s3.us-east-2.amazonaws.com"
}

in/outPlugInHookTags

• Generate lambda, topics, iam policies, and other resources from listed plugInHookTags
• The main project has the full schema, both projects generate from that single schema
• inPlugInHookTags for main project, and outPlugInHookTags for sub project generate topic and flows for receiving requests
• outPlugInHookTags for main project, and inPlugInHookTags for sub project generate topic for receiving response and triggered lambda inside flows where the plugInHookTags are invoked from
• in and out plugInHookTags are optional, if have both is a bidirectional plugIn

Params sent to sub project when creating a connection

• if the sub project's base objType has requiredOnCreate fields, they must all be sent according to that objects schema
• additional fields can be sent using sendFieldNamesOnCreate property
• sendFieldNamesOnCreate can come from the base objType or be custom user inputted values
• frontend shows required params when creating plugin connection

ObjType Connection

• a plugIn always connects with one objType in each project, never more, is 1-to-1
• plugIn requests between projects always send an identifier of the source objType and externalIdentifiers so the receiving project can identify the connection
• object schema lists which plugInTags can be connected with it's instances

PlugIn Hooks

• flowStep in flow schema lists which plugInHookTags are invoked from that flowStep, but developer must add code where plugInHookTag is used and prepare eg identifiers
• flow schema not need to have a main objType or match plugIn's objType
• flowStep config sets which objType each plugInHook uses, matches objType in plugIn schema

submittedBy userId

• submittedBy userId is passed between plugIn projects
• see User_Data_Handling#Per_Project_User_Data

Configuring Connection

• each project has object level RBAC permission for each plugIn
• RBAC is connected to the main project's objType, or it's belongTo objType
• example Commerce initiates the plugIn to comAcc, so the object level RBAC is in the Business service for Business objType
• if a plugIn's main objType is Commerce.Inventory, then the RBAC will be in the Business service for Business objType, because Inventory belongsTo Business
• in main project if user has permission, can send request to create new connection using an existing instance of the plugIn's main objType
• receiving/sub project must have reference plugIn schema matching main project

PlugIn Creates Instance

• the main service must always have an instance of it's objType to initialize a plug in connection
• the sub project optionally has an instance depending on plugInSchema createInstance setting

createInstance = false

• creates plugIn connection between two existing instances
• sub project (externalConfig) objType identifiers must be sent as well and must exist on receiving project
• user must have permission on receiving project to complete plugIn connection for the sub project objType
• might not need to send targetIdentifiers (belongTo for sub project object instance), can query on sub project before completing the connection, checking submittedBy userId has permission to make the connection

RBAC for creating connection on receiving project

• sub project objType must have rbac addOnDataStructure
• sub project objType has object level permission per plugIn whether can receive create plugIn connection from submittedBy user

createInstance = true

• main project has existing instance, create new instance on sub project
• (maybe force sub project objType to have rbac, not belongTo) if the external objType belongsTo, targetIdentifiers must be sent when creating new connection
• new sub project objType instance is created, including RBAC and AccountLimits for the submitted by userId
• user must then setup RBAC roles for any other users that are using the plugIn
• objType in sub project must be generated by system, eg: Organization in Accounting can be created by plugIn but also by user, currently must set schema to generated by system only

RBAC for creating connection on receiving project

• no rbac needed, a new object is created
• external project is protected from dishonest connections using iam policy that only allows authorized projects to send requests to create connections

example

• Commerce has ComAcc plugIn and does not create Business instance
• ComAcc has Commerce plugIn and does create ComAccBusiness instance
• ComAcc has Accounting plugIn and does not create ComAccBusiness instance
• Accounting has ComAcc plugIn and does create Organization instance

CreatePlugInConnection flow

• each plugIn generates flowSchemas for creating a plugIn connection
• in the service for base objType for both projects
• main project flow is user initiated
• sub project flow is system initiated (by plugIn message from main project)
• if createInstance is false must send/receive externalIdentifiers also

Rbac and limits

• create connection flow in main project has objType of base objType
• this creates object level rbac and account limit settings for creating plugIn connections

plugInConnection Data

• When a plugIn connection is made a record is saved into the both project's in the service matches it's plugIn objType
• (optional) in future could check plugIn schema, only need this table on project that has plugInHooks, one direct plugIns only need on one side
• objTypeConcat might not be required because dynamo table is per service, but risk two objects in same service have chance of matching identifiers so add

{
   plugInConnectionId: "objTypeConcat_identifiersConcat",
   plugInTag: "xxx",
   externalIdentifiers: {
      xx: "yy",
   }
}

plugInHook request

Query plugInConnection

• flowSteps list which plugInHooks will be checked within it's code, and what base objType that hook does work on
• When the code reaches a plugInHookTag it queries plugInConnection table
• tablename is hardcoded
• service where dynamo is located is found from flowstep>plugInHooks>objType>serviceTag
• identifiers must be prepared beforehand in logic
• query returns all object instance + plugInTag records that connect to that hook's objType

Send Request to plugIn

• iterate each plugInConnection found
• get plugIn schema from S3 (memoize)
• check if this plugIn has current plugInHook, if not do nothing
• check plugIn schema invocationType for current plugInHook
• send request to external project
• sns topic for invocationType=async, apiGateway for invocationType=sync
• send identifiers for objType from sending side (used to query plugInConnection) and externalIdentifiers (from plugInConnection record)

Recieve plugIn Request

• because plugIn connections are 1-to-1 can use externalIdentifiers to find plugIn connection if needed
• (maybe force not have belongTo) if externalIdentifiers on receiving project has belongTo, can follow that to get targetIdentifiers, so sending project not need to send these

RBAC for Receiving Project

• No rbac is required for plugInHook logic on the receiving project
• Is assumed if the user has permission for that action on the sending project, then they have permission on the receiving project

Common Concepts

PlugIn Project

• connects two projects together
• maintains configuration settings that are specific to the connection
• handles flows specific to the connection
• The two connected projects have as little linkage code or data structures as possible, abstract this into the plugin project
• possible that a plugIn project could connect multiple projects, eg a logging or analysis plugin, each is a separate plugIn

Link Objects

• when object in one project connects with an object in another another project a plugin hook connects to the plugin project which records the per instance links
• required when future flows require knowledge of previous links

Link Configs

• configuration for the plugIn project is handled within the plugIn's own interface
• eg setting which Accounting Accounts connect with which Commerce initiated flows, such as transactions, purchase orders, sales
• plugIn Project has it's own data with submittedBy, versionData, RBAC etc..

PlugIn Project RBAC

• Data and flows that are handled within one project have rbac applied to them
• for example link configs in a plugIn project, user must setup rbac in the plugIn project to allow other users to configure link configs

Plugin Connection Blocks Data Updates

• It might be required that when a plugin connection flows are blocked from proceeding as they can only be initiated from the plugin
• eg once an Accounting Organization is linked to Commerce it might not be possible to add entries directly through the Accounting interface
• can code this into flows where the plugIn hook returns a switch that throws an error in the sending project
• eg: when Accounting Organization is linked to ComAcc add a plugInHook to create Ledger flow which sends request to ComAcc which throws error, because should never get a request from Accounting to create a Ledger, it should come from Commerce

Working documents

PlugIn Project Integration

Overview

Creating connections between projects with code hooks that send data between them.

Terminology

• Main project: Project the has the full plugIn schema and initiates connections
• Sub project: Project that has the reference plugIn schema
• Sending project: Project that has a plugInHook and sends request to another project
• Receiving project: Project that receives the request from the sending project
• Base objType: The obType set in plugIn schema, both sides of the plugIn have a base objType, the main project is the top level objType in plugIn schema, for the sub project it is the externalConfig objType

PlugIns can be bi-directional, so the main and sub projects can be both sending and receiving projects.

Structure

PlugIn Schema

• each project manually adds available plugIns in an S3 bucket
• plugIn is identified by a plugInTag
• a plugIn connects only one project to another project
• if multiple projects can connect to a single project seperate each as it's own plugin with it's own lambdas/flows, shared code in libs

PlugInHookTags

• plugInHookTags are defined by the code in that project
• plugIns choose which plugInHookTags that want to hook into
• for this reason we can identify plugInHooks by their plugInHookTag only, within one project each plugInHookTag must be unique
• the receiving project will also generate websocket actions/api endpoints/flows and lambdas according to the sending projects plugInHookTags
• the receiving project's generated resources also include the serviceTag which comes from the other project's plugInSchema objType
• so different plugIns that send to the same project>service>objType must be careful not to duplicate plugInHookTag names

Schema files

• both projects for the plugIn have an S3 schema file for a plugIn
• the main project has the full schema
• the sub project lists a reference to the main project's schema
• each plugIn has it's own file in the projects schema, filename same plugInTag

Example Core plugIn Schema

{
	objType:{
		serviceTag: "Business",
		objectType: "business"
	},
    inPlugInHookTags:{
		// same structure as outPlugInHookTags
    },
    outPlugInHookTags:{
      xxx: { // plugInHookTag
        invocationType: "async",
      },
      yyy: {
        invocationType: "sync",
      }
    },
	externalConfig:{
		createInstance: true, // setting of external createInstance , internal 
		objType:{
		serviceTag: "ComAcc",
		objectType: "comAccLinkByBusiness"
		},
		sendFieldNamesOnCreate: [
			dailyRateCurrencyTableId: { // name of param sent to sub project
				source: "baseObject", // comes from the main projects base objType
				fieldName: "dailyRateCurrencyTableId",
				canEdit: true // default false, whether user can adjust value when making plugin connection, if false forced same as base object
			},
			functionalCurrency: { // name of param sent to sub project
				source: "baseObject", // comes from the main projects base objType
				fieldName: "functionalCurrency"
			},
			precision: { // name of param sent to sub project
				source: "baseObject", // comes from the main projects base objType
				fieldName: "precision"
			},
			customField: {
				source: "custom", // similar to objType fieldNames, but only basic properties
				type: "string",
				validation: {
					pattern: pattern
				},		
			},
		],
		schemaBucketUrl: "object-schema.s3.us-east-2.amazonaws.com"
		snsArnPrefix: "arn:aws:sns:us-east-2:860226617901:",
		//      websocketUrl: "wss://kw86izcde3.execute-api.us-east-2.amazonaws.com/"
		apiBaseUrl: "https://1y9v1t7rvc.execute-api.us-east-2.amazonaws.com/" // for full endpoint add: /{serviceTag}/{plugInTag}_{plugInHookTag}
    }
}

Example Reference plugIn Schema

{
   "schemaBucketUrl": "object-schema.s3.us-east-2.amazonaws.com"
}

in/outPlugInHookTags

• Generate lambda, topics, iam policies, and other resources from listed plugInHookTags
• The main project has the full schema, both projects generate from that single schema
• inPlugInHookTags for main project, and outPlugInHookTags for sub project generate topic and flows for receiving requests
• outPlugInHookTags for main project, and inPlugInHookTags for sub project generate topic for receiving response and triggered lambda inside flows where the plugInHookTags are invoked from
• in and out plugInHookTags are optional, if have both is a bidirectional plugIn

ObjType Connection

• a plugIn always connects with one objType in each project, never more, is 1-to-1
• plugIn requests between projects always send an identifier of the source objType and externalIdentifiers so the receiving project can identify the connection
• object schema lists which plugInTags can be connected with it's instances

PlugIn Hooks

• flowStep in flow schema lists which plugInHookTags are invoked from that flowStep, but developer must add code where plugInHookTag is used and prepare eg identifiers
• flow schema not need to have a main objType or match plugIn's objType
• flowStep config sets which objType each plugInHook uses, matches objType in plugIn schema

submittedBy userId

• submittedBy userId is passed between plugIn projects
• see User_Data_Handling#Per_Project_User_Data

Configuring Connection

• each project has object level RBAC permission for each plugIn
• RBAC is connected to the main project's objType, or it's belongTo objType
• example Commerce initiates the plugIn to comAcc, so the object level RBAC is in the Business service for Business objType
• if a plugIn's main objType is Commerce.Inventory, then the RBAC will be in the Business service for Business objType, because Inventory belongsTo Business
• in main project if user has permission, can send request to create new connection using an existing instance of the plugIn's main objType
• receiving/sub project must have reference plugIn schema matching main project

PlugIn Creates Instance

• the main service must always have an instance of it's objType to initialize a plug in connection
• the sub project optionally has an instance depending on plugInSchema createInstance setting

createInstance = false

• creates plugIn connection between two existing instances
• sub project (externalConfig) objType identifiers must be sent as well and must exist on receiving project
• user must have permission on receiving project to complete plugIn connection for the sub project objType
• might not need to send targetIdentifiers (belongTo for sub project object instance), can query on sub project before completing the connection, checking submittedBy userId has permission to make the connection

RBAC for creating connection on receiving project

• sub project objType must have rbac addOnDataStructure
• sub project objType has object level permission per plugIn whether can receive create plugIn connection from submittedBy user

createInstance = true

• main project has existing instance, create new instance on sub project
• (maybe force sub project objType to have rbac, not belongTo) if the external objType belongsTo, targetIdentifiers must be sent when creating new connection
• new sub project objType instance is created, including RBAC and AccountLimits for the submitted by userId
• user must then setup RBAC roles for any other users that are using the plugIn
• objType in sub project must be generated by system, eg: Organization in Accounting can be created by plugIn but also by user, currently must set schema to generated by system only

RBAC for creating connection on receiving project

• no rbac needed, a new object is created
• external project is protected from dishonest connections using iam policy that only allows authorized projects to send requests to create connections

example

• Commerce has ComAcc plugIn and does not create Business instance
• ComAcc has Commerce plugIn and does create ComAccBusiness instance
• ComAcc has Accounting plugIn and does not create ComAccBusiness instance
• Accounting has ComAcc plugIn and does create Organization instance

CreatePlugInConnection flow

• each plugIn generates flowSchemas for creating a plugIn connection
• in the service for base objType for both projects
• main project is user initiated
• sub project is system initiated (by plugIn message from main project)
• if createInstance is false must send/receive externalIdentifiers also

Rbac and limits

• create connection flow in main project has objType of base objType
• this creates object level rbac and account limit settings for creating plugIn connections

plugInConnection Data

• When a plugIn connection is made a record is saved into the both project's in the service matches it's plugIn objType
• (optional) in future could check plugIn schema, only need this table on project that has plugInHooks, one direct plugIns only need on one side
• objTypeConcat might not be required because dynamo table is per service, but risk two objects in same service have chance of matching identifiers so add

{
   plugInConnectionId: "objTypeConcat_identifiersConcat",
   plugInTag: "xxx",
   externalIdentifiers: {
      xx: "yy",
   }
}

plugInHook request

Query plugInConnection

• flowSteps list which plugInHooks will be checked within it's code, and what base objType that hook does work on
• When the code reaches a plugInHookTag it queries plugInConnection table
• tablename is hardcoded
• service where dynamo is located is found from flowstep>plugInHooks>objType>serviceTag
• identifiers must be prepared beforehand in logic
• query returns all object instance + plugInTag records that connect to that hook's objType

Send Request to plugIn

• iterate each plugInConnection found
• get plugIn schema from S3 (memoize)
• check if this plugIn has current plugInHook, if not do nothing
• check plugIn schema invocationType for current plugInHook
• send request to external project
• sns topic for invocationType=async, apiGateway for invocationType=sync
• send identifiers for objType from sending side (used to query plugInConnection) and externalIdentifiers (from plugInConnection record)

Recieve plugIn Request

• because plugIn connections are 1-to-1 can use externalIdentifiers to find plugIn connection if needed
• (maybe force not have belongTo) if externalIdentifiers on receiving project has belongTo, can follow that to get targetIdentifiers, so sending project not need to send these

RBAC for Receiving Project

• No rbac is required for plugInHook logic on the receiving project
• Is assumed if the user has permission for that action on the sending project, then they have permission on the receiving project

Common Concepts

PlugIn Project

• connects two projects together
• maintains configuration settings that are specific to the connection
• handles flows specific to the connection
• The two connected projects have as little linkage code or data structures as possible, abstract this into the plugin project
• possible that a plugIn project could connect multiple projects, eg a logging or analysis plugin, each is a separate plugIn

Link Objects

• when object in one project connects with an object in another another project a plugin hook connects to the plugin project which records the per instance links
• required when future flows require knowledge of previous links

Link Configs

• configuration for the plugIn project is handled within the plugIn's own interface
• eg setting which Accounting Accounts connect with which Commerce initiated flows, such as transactions, purchase orders, sales
• plugIn Project has it's own data with submittedBy, versionData, RBAC etc..

PlugIn Project RBAC

• Data and flows that are handled within one project have rbac applied to them
• for example link configs in a plugIn project, user must setup rbac in the plugIn project to allow other users to configure link configs

Locking Data

• It might be required that when a plugin connection flows are blocked from proceeding as they can only be initiated from the plugin
• eg once an Accounting Organization is linked to Commerce it might not be possible to add entries directly through the Accounting interface
• can code this into flows where the plugIn hook returns a switch that throws an error in the sending project
• eg: when Accounting Organization is linked to ComAcc add a plugInHook to create Ledger flow which sends request to ComAcc which throws error, because should never get a request from Accounting to create a Ledger, it should come from Commerce

Working documents

PlugIn Project Integration