Data Models

Overview

Complex data models with their own coded libraries to enforce data rules.

Timestamps use millisecond precision, connective timestamps (that create a series of periods) differ by 1 millisecond.

Ledgers

• Maintains a strict history of changes using a entries
• During it's open life (until closed) base object can only have one entry instance attached to it for each ledger (or per groupingRelationship per ledger)
• ledger entries for one ledger cannot overlap, differ by 1 millisecond
• while open, ledger will have one current entry that does not have "to" timestamp, when a new entry is added onld entry link changes to old with "to" timestamp and a new entry is created
• time of each entry is according to when the graph query actually updates the database record
• addLedgerEntry for a non-existent/closed ledger is an error, must specifically openLedger (or makeEntryOnCreate) to start adding entries

Terminology

• Base object: The object instance that has ledger applied, all entries attach to this object
• Target object: The object that changes for each ledger entry (created or existing depending on entryObject setting)
• Entry: One setting for the ledger
• Entry object: entryObject = true only, node created between base and target objects

addOnDataStructure

{
	addOnDataStructure: [
		{
			type: "ledger",
			prefix: "xxx",
			entryObject: true, //boolean, if true creates new entry object with each entry, false creates relationships to existing objects
			groupByPathLinkType: { // optional, allows one ledger for each link that exists from base Object, only useful when link is "to many"
				objType: {
					serviceTag: "xxx",
					objectType: "yyy"
				},
				relType: {
					serviceTag: "xxxx",
					relationshipTag: "yyyy"
				},
				relationshipDirection: "to"
			}
			storageResourceTag: "xxx", // tag of graph where ledger is recorded
			targetObjType: {}, // if movable, this is the default objType an entry is made for
			makeEntryOnCreate: true, // whether an entry is created at same time as creating base object
			canClose: true, // whether can be closed
			canOpenAfterClosed: true, // whether can be opened again after being closed (only used when canClose = true)
	
			// movable ledger
			moveToTargetObjTypes: [
				{}, // list of other objTypes this ledger can point to
			],
			
		},
	],
}

entryObject true

• an entry object (separate to target object) is created with each entry
• to and from timestamps are stored in entry object
• entryObject is a link node between the base object and an existing target object
• new target object instance must already exist when creating a new entry
• many base objects can point to one target object using their own entry object instances
• example: unit's ledger connecting to stock locations

entryObject false

• a new target object instance is created with each entry
• to and from timestamps are stored in each target object
• example: unitTrackedProperty

openLedger and closeLedger endpoints

• if makeEntryOnCreate = false or canOpenAfterClosed = true has openLedger endpoint
• if canClose = true has closeLedger endpoint

groupByPathLinkType

• not yet code for makeEntryOnCreate = true
• maybe not yet code for entryObject = true, needs to go check existing current entry two steps: targetObject > groupByPathLinkType instance
• openLedger validates identifiers for groupByPathLinkType.objType has existing instance
• openLedger validates no existing current entry between base object and groupByPathLinkType object Instance

Cases

Forever/Always Ledger

• makeEntryOnCreate = true
• canClose = false
• canOpenAfterClosed = (unused because canClose = false)

• Not need to code for this yet, no use cases (but can keep existing code)
• Creates an entry at the time of creating the base object with a relationship linking them
• no openLedger or closeLedger end points

example

• none yet

Movable Ledger

• makeEntryOnCreate = true
• canClose = true
• canOpenAfterClosed = true
• moveToTargetObjTypes = {..}

• Creates an entry at the time of creating the base object with a relationship linking them
• Can move between targetObjTypes
• can close, eg when Unit no longer has stock
• can open, eg when unit is put back into stock

example: Unit Location

• can move into Packaging which handles it's own entries
• can move back out to it's own location
• when in packaging the moveToTargetObjType ledger is the link between the Unit and the UnitPackaging
• UnitPackaging has it's own ledger pointing to location
• When outside of packaging the ledger can update to new locations
• When inside packaging the ledger can update which packaging unit it is currently inside (moves between packaging units without having to return to a unitStandard>location entry

Transitory Ledger

• makeEntryOnCreate = true
• canClose = true
• canOpenAfterClosed = false

• Not need to code for this yet, no use cases
• Creates an entry at the time of creating the base object with a relationship linking them
• Can be closed
• Does not allow new ledgers, once closed stays closed

example

• none yet

Transient Ledger

• makeEntryOnCreate = false
• canClose = true
• canOpenAfterClosed = true
• groupByPathLinkType = {..}

• Entry not created when creating base object
• Can be closed
• Can be opened
• Only one ledger can exist at any time, if try to open a new ledger when one exists is error
• If try to closeLedger when already closed is error

example: Unit Tracked Property

• not created when making UnitStandard instance, create UnitStandard code (or frontend) must create entries for each tracked property after unitStandard
• do this because UnitTypeTrackedProperties can be added and removed from parent UnitType, so need method of creating new ledgers after creation of UnitStandard
• groupByPathLinkType is per isUnitTypeTrackedProperty relationship
• ledger can be closed when a UnitTypeTrackedProperty is removed from the parent UnitType (but maybe leave it hanging so not need to dig all child Units as the last value of an unused UnitTypeTrackedProperty)
• validation that UnitTypeTrackedProperty belongs to UnitStandard's UnitType must be done before sending to graph openLedger endpoint
• makeEntryOnCreate = false means has openLedger endpoint

Data Elements

LedgerEntry ObjInstance

• entryObject = true
• Each entry creates a new link object instance
• Records from and to timestamps (only from timestamp when is the current entry)

currentEntry Relationship

• only one exists per base object per ledger

oldEntry Relationship

• many exist
• moved from currentEntry when addLedgerEntry or closeLedger

Graph Endpoints

OpenLedger

• Only used for Transient Ledger, so not code yet
• Starts a ledger with new entry
• Validates no existing ledger exists

AddLedgerEntry

• move currentEntry relationship to oldEntry
• create new LedgerEntry with currentEntry relationship
• ensure timestamps connect
• optional moveTo setting for MovableLedger

CloseLedger

• move currentEntry relationship to oldEntry
• add to Timestamp to entry

Relationship Ledger

• moveRelationship .. ?

Reserve Quantity

• ensures quantity is not double-reserved
• stock is reserved and success returned to calling flow, calling flow does it's work (eg multiple reservations all pass), flow sends confirm or cancel request to finalize the usage
• multiple requests are placed in a queue and all will be processed once, one by one
• queue works the same as lockUpdate, but adds additional standard code for reserving and cancel/confirm reservations

lockUpdate

• for critical object updates that need to be processed but must process one update at a time
• all requests will be processed in the order they are received by the flow/s
• multiple flows can be controlled by one lockUpdate prefix
• object schema addOnDataStructure sets lock prefixes, one objType can have multiple lock prefixes, eg: for independant protection of different data
• applicable flow schemas set which lockUpdate prefix they rely on
• flow checks if lock is open, if not the request is placed into LockQueue for processing

LockQueue

• DynamoDB table
• is a list of all requests remaining to process
• partition key includes instance identifier
• queue is per identifier, so one object instance ultimately has it's own queue
• one instance's lock does not affect other instances
• cannot use SQS because does not scale for one queue per instance

LockHistory

• DynamoDB table
• records all requests and their status, at the time they begin processing
• records a unique identifier for each request that can be sent by external flow to ensure only this request is completed and completed once
• protects against double processing in case where main record status and unique request id are removed and lambda halts and restarts
• after a request finishes it checks LockQueue for remaining task, without LockHistory there is a chance a queued request might be handled twice

TTL on LockHistory

• after request completes should keep this record for a while in case completion request arrives multiple times (although could perhaps handle no record as an already completed request)
• could add a TTL to the record to maintain record a reasonable time and then auto remove
• a Lambda could be triggered when TTL happens which checks the removed records status, if it has not yet been processed could consider a failed request that can be cleaned up so does not block other requests

Relationship Switch

• uses changeRelationship to ensure always has a setting
• uses originTimestamp to ensure the last clicked/submitted setting from the frontend is applied

Attribute Tree

• creates a hierarchy of attribute values
• all values have a label
• link nodes in the tree are a child of the base object or another link node
• values are shared anywhere in the tree hierarchy, and across different trees
• labels and values have translations

Attribute Values

• attribute values use Manager/Handler services
• most values are just a value, eg string or numeric
• but handler services allows for per use case special value types
• eg: currency values that specify a currency

Working documents

Data Models