Service - Import Data

Process

ProcessPendingLinks:

1. count DynamoDB records for PendingObjectMain and PendingLinks and save in ImportBatchMain table
2. iterate all PendingLink records:
   • check referenceId has a valid record in PendingObjectReference
   • check linkTag is valid (objectId is the child, referenceId object is the parent)
   • if linkType is Dependent:
     • if separateDependentLinkCreate false and child exist add error
     • save (DependentPendingObject)awaitingMultipleSteps for the child object, waiting for the parent object to be created (multiple so can handle multiple parents for this child)
     • save (DependentPendingLink)awaitingStep for the PendingLink, waiting for the child object to be created
   • if linkType is Independent, save (IndependentPendingLink)awaitingMultipleSteps for the PendingLink, waiting for both child and parent to be created
   • save this PendingLink into (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
3. if any errors found prior to this step stop processing and mark feed as status error, remove any saved awaitingMultipleSteps

ProcessPendingObjects:

1. iterate all PendingObjectMain records
2. save this PendingObjectMain into (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
3. iterate any PendingLinks that have this object as the child (use Dynamo startsWith on pendingLinkId field)
   • if linkType is Dependent, stop processing this PendingObjectMain record (already have awaitingMultipleSteps saved)
4. if PendingObjectMain exist:
   • set status of PendingObjectMain to "complete"
   • do PendingObjectComplete lib
5. if PendingObjectMain not exist:
   • set status of PendingObjectMain to "creating"
   • send to external service to create

After external service creates object:

1. sets status of PendingObjectMain to "complete", add object identifier ids to PendingObjectMain record
   • • do PendingObjectComplete lib

PendingObjectComplete lib:

1. remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
2. iterate any Dependent (DependentPendingObject)awaitingMultipleSteps
   • find the child object that was waiting this parent
   • check if that child object has any remaining (DependentPendingObject)awaitingMultipleSteps, if no then:
     • find all Dependent pendingLinks for this child object (using Dynamo BeginsWith)
     • if child object exist:
       • set status of child object to "complete"
       • do PendingObjectComplete lib
     • if child object not exist:
       • send message to external service to create child object
       • include in message to external service to create child object all parent object identifier ids
       • set status of child PendingObjectMain to "creating"
3. iterate any Dependent PendingLink (DependentPendingLink)awaitingStep
   • if pendingLink separateDependentLinkCreate = false(default):
     • sets status of PendingLink to "complete"
     • remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain
   • if pendingLink separateDependentLinkCreate = true:
     • set status of PendingLink to "creating"
     • send to external service to create link, including object identifier ids of both objects
4. iterate any Independent (IndependentPendingLink)awaitingMultipleSteps
   • check any remaining awaitingMultipleSteps for that PendingLink, if no then:
     • set status of PendingLink to "creating"
     • send to external service to create link, including object identifier ids of both objects

After external service creates link:

1. sets status of PendingLink to "complete"
2. remove (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain

When removing (ImportBatchMain)awaitingMultipleSteps for ImportBatchMain (both PendingObjectMain and PendingLink)

1. check if any remain, if not then set ImportBatchMain to complete and send ImportBatchMainComplete message