Sven the Barbarian at 14:39, 28 February 2023

2023-02-28T14:39:07Z

← Older revision		Revision as of 14:39, 28 February 2023
Line 3:		Line 3:
	= Process =		= Process =

	ProcessPendingLinks:		== ProcessPendingLinks ==

	# count DynamoDB records for PendingObjectMain and PendingLinks and save in ImportBatchMain table		# count DynamoDB records for PendingObjectMain and PendingLinks and save in ImportBatchMain table
Line 17:		Line 17:
	# if any errors found prior to this step stop processing and mark feed as status error, remove any saved awaitingMultipleSteps		# if any errors found prior to this step stop processing and mark feed as status error, remove any saved awaitingMultipleSteps

	ProcessPendingObjects:		== ProcessPendingObjects ==

	# iterate all PendingObjectMain records		# iterate all PendingObjectMain records
Line 30:		Line 30:
	#* send to external service to create		#* send to external service to create

	After external service creates/checks object exists:		== After external service creates/checks object exists ==

	# query (ImportBatchMain)awaitingMultipleSteps using pendingObjectId (saved in correlation ids) to find importBatchId		# query (ImportBatchMain)awaitingMultipleSteps using pendingObjectId (saved in correlation ids) to find importBatchId
Line 61:		Line 61:
	# remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain for the completed object, do this last because use AwaitingMultipleSteps table to find object at start of this Lambda		# remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain for the completed object, do this last because use AwaitingMultipleSteps table to find object at start of this Lambda

	After external service creates link:		== After external service creates link ==

	# sets status of PendingLink to "complete"		# sets status of PendingLink to "complete"
	# remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain		# remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain

	When removing (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain (both PendingObjectMain and PendingLink)		== When removing (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain (both PendingObjectMain and PendingLink) ==

	# check if any remain, if not then set ImportBatchMain to complete and send ImportBatchMainComplete message		# check if any remain, if not then set ImportBatchMain to complete and send ImportBatchMainComplete message

Sven the Barbarian: Created page with "Service - Import Data = Process = ProcessPendingLinks: # count DynamoDB records for PendingObjectMain and PendingLinks and save in ImportBatchMain table # iterate all PendingLink records: #* check referenceId has a valid record in PendingObjectReference #* check linkTag is valid (pendingObjectId is the child, referenceId object is the parent) #* if linkType is Dependent: # if separateDependentLinkCreate false and child exist add error # save (DependentPendingO..."

2023-02-28T13:52:59Z

Created page with "Service - Import Data = Process = ProcessPendingLinks: # count DynamoDB records for PendingObjectMain and PendingLinks and save in ImportBatchMain table # iterate all PendingLink records: #* check referenceId has a valid record in PendingObjectReference #* check linkTag is valid (pendingObjectId is the child, referenceId object is the parent) #* if linkType is Dependent: #** if separateDependentLinkCreate false and child exist add error #** save (DependentPendingO..."

New page

[[Service - Import Data]]

= Process =

ProcessPendingLinks:

# count DynamoDB records for PendingObjectMain and PendingLinks and save in ImportBatchMain table
# iterate all PendingLink records:
#* check referenceId has a valid record in PendingObjectReference
#* check linkTag is valid (pendingObjectId is the child, referenceId object is the parent)
#* if linkType is Dependent:
#** if separateDependentLinkCreate false and child exist add error
#** save (DependentPendingObject)awaitingMultipleSteps for the child object, waiting for the parent object to be created (multiple so can handle multiple parents for this child)
#** save (DependentPendingLink)awaitingStep for the PendingLink, waiting for the child object to be created
#* if linkType is Independent, save (IndependentPendingLink)awaitingMultipleSteps for the PendingLink, waiting for both child and parent to be created
#* save this PendingLink into (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
# if any errors found prior to this step stop processing and mark feed as status error, remove any saved awaitingMultipleSteps

ProcessPendingObjects:

# iterate all PendingObjectMain records
# save this PendingObjectMain into (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
# iterate any PendingLinks that have this object as the child (use Dynamo startsWith on pendingLinkId field)
#* if linkType is Dependent, stop processing this PendingObjectMain record (already have awaitingMultipleSteps saved)
# if feed set PendingObjectMain already exist (has identifier property):
#* set status of PendingObjectMain to "checking"
#* send to external service to check exist
# if feed set PendingObjectMain not already exist (has fields property):
#* set status of PendingObjectMain to "creating"
#* send to external service to create

After external service creates/checks object exists:

# query (ImportBatchMain)awaitingMultipleSteps using pendingObjectId (saved in correlation ids) to find importBatchId
# sets status of PendingObjectMain to "complete", add object identifier ids to PendingObjectMain record (exist will already have, maybe validate)
# iterate any Dependent (DependentPendingObject)awaitingMultipleSteps
#* find the child object that was waiting this parent
#* check if that child object has any remaining (DependentPendingObject)awaitingMultipleSteps, if no then:
#** if child object exist:
#*** set status of child object to "checking"
#*** send to external service to check exist
#** if child object not exist:
#*** find all Dependent pendingLinks for this child object (using Dynamo BeginsWith) so can add parent objects to message
#*** send message to external service to create child object
#*** include in message to external service to create child object all parent object identifier ids/object type/link tags
#*** set status of child PendingObjectMain to "creating"
#* remove Dependent (DependentPendingObject)awaitingMultipleSteps
# iterate any Dependent PendingLink (DependentPendingLink)awaitingStep
#* if pendingLink separateDependentLinkCreate = false(default):
#** sets status of PendingLink to "complete"
#** remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
#* if pendingLink separateDependentLinkCreate = true:
#** set status of PendingLink to "creating"
#** send to external service to create link, including object identifier ids of both objects
#* remove Dependent PendingLink (DependentPendingLink)awaitingStep
# iterate any Independent (IndependentPendingLink)awaitingMultipleSteps
#* check any remaining awaitingMultipleSteps for that PendingLink, if no then:
#** set status of PendingLink to "creating"
#** send to external service to create link, including object identifier ids of both objects
#* remove Independent (IndependentPendingLink)awaitingMultipleSteps
# remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain for the completed object, do this last because use AwaitingMultipleSteps table to find object at start of this Lambda

After external service creates link:

# sets status of PendingLink to "complete"
# remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain

When removing (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain (both PendingObjectMain and PendingLink)

# check if any remain, if not then set ImportBatchMain to complete and send ImportBatchMainComplete message

[[Category:Working documents| 2023-02-28]]
[[Category:Working documents - Import Data| 2023-02-28]]

2023-02-28 - ImportData Process - Revision history

Sven the Barbarian at 14:39, 28 February 2023