2023-02-28 - ImportData Process

Service - Import Data

Process

ProcessPendingLinks

1. count DynamoDB records for PendingObjectMain and PendingLinks and save in ImportBatchMain table
2. iterate all PendingLink records:
   • check referenceId has a valid record in PendingObjectReference
   • check linkTag is valid (pendingObjectId is the child, referenceId object is the parent)
   • if linkType is Dependent:
     • if separateDependentLinkCreate false and child exist add error
     • save (DependentPendingObject)awaitingMultipleSteps for the child object, waiting for the parent object to be created (multiple so can handle multiple parents for this child)
     • save (DependentPendingLink)awaitingStep for the PendingLink, waiting for the child object to be created
   • if linkType is Independent, save (IndependentPendingLink)awaitingMultipleSteps for the PendingLink, waiting for both child and parent to be created
   • save this PendingLink into (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
3. if any errors found prior to this step stop processing and mark feed as status error, remove any saved awaitingMultipleSteps

ProcessPendingObjects

1. iterate all PendingObjectMain records
2. save this PendingObjectMain into (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
3. iterate any PendingLinks that have this object as the child (use Dynamo startsWith on pendingLinkId field)
   • if linkType is Dependent, stop processing this PendingObjectMain record (already have awaitingMultipleSteps saved)
4. if feed set PendingObjectMain already exist (has identifier property):
   • set status of PendingObjectMain to "checking"
   • send to external service to check exist
5. if feed set PendingObjectMain not already exist (has fields property):
   • set status of PendingObjectMain to "creating"
   • send to external service to create

After external service creates/checks object exists

1. query (ImportBatchMain)awaitingMultipleSteps using pendingObjectId (saved in correlation ids) to find importBatchId
2. sets status of PendingObjectMain to "complete", add object identifier ids to PendingObjectMain record (exist will already have, maybe validate)
3. iterate any Dependent (DependentPendingObject)awaitingMultipleSteps
   • find the child object that was waiting this parent
   • check if that child object has any remaining (DependentPendingObject)awaitingMultipleSteps, if no then:
     • if child object exist:
       • set status of child object to "checking"
       • send to external service to check exist
     • if child object not exist:
       • find all Dependent pendingLinks for this child object (using Dynamo BeginsWith) so can add parent objects to message
       • send message to external service to create child object
       • include in message to external service to create child object all parent object identifier ids/object type/link tags
       • set status of child PendingObjectMain to "creating"
   • remove Dependent (DependentPendingObject)awaitingMultipleSteps
4. iterate any Dependent PendingLink (DependentPendingLink)awaitingStep
   • if pendingLink separateDependentLinkCreate = false(default):
     • sets status of PendingLink to "complete"
     • remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
   • if pendingLink separateDependentLinkCreate = true:
     • set status of PendingLink to "creating"
     • send to external service to create link, including object identifier ids of both objects
   • remove Dependent PendingLink (DependentPendingLink)awaitingStep
5. iterate any Independent (IndependentPendingLink)awaitingMultipleSteps
   • check any remaining awaitingMultipleSteps for that PendingLink, if no then:
     • set status of PendingLink to "creating"
     • send to external service to create link, including object identifier ids of both objects
   • remove Independent (IndependentPendingLink)awaitingMultipleSteps
6. remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain for the completed object, do this last because use AwaitingMultipleSteps table to find object at start of this Lambda

After external service creates link

1. sets status of PendingLink to "complete"
2. remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain

When removing (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain (both PendingObjectMain and PendingLink)

1. check if any remain, if not then set ImportBatchMain to complete and send ImportBatchMainComplete message