2025-11-24 ImportData Document - Revision history

Sven the Barbarian at 09:32, 31 December 2025

2025-12-31T09:32:05Z

← Older revision		Revision as of 09:32, 31 December 2025
Line 1:		Line 1:
	= Process ImportData =		= Process ImportData =

	== ProcessCsvFile ==		== ProcessCsvFile ==

	#Receive bucketName and fieldName.		#Receive bucketName and fieldName.
	#Define the range e.g. range = "bytes=0-5000".		#Define the range e.g. range = "bytes=0-5000".
	#Fetch data from S3 with:<syntaxhighlight lang="javascript">		#Fetch data from S3 with:
			<syntaxhighlight lang="javascript">
	{		{
	Bucket: 'izara-importdata',		Bucket: 'izara-importdata',
Line 10:		Line 13:
	}		}
	</syntaxhighlight>		</syntaxhighlight>

	#Take the importBatchId from the S3 metadata and use it to get ImportBatchMain to retrieve importConfigId.		#Take the importBatchId from the S3 metadata and use it to get ImportBatchMain to retrieve importConfigId.
	#Use the retrieved importConfigId to get CsvImportConfig.		#Use the retrieved importConfigId to get CsvImportConfig.
Line 19:		Line 23:

	== ProcessRowData ==		== ProcessRowData ==

	#Receive importBatchId.		#Receive importBatchId.
	#Use the received importBatchId to get ImportBatchMain and extract importConfigId.		#Use the received importBatchId to get ImportBatchMain and extract importConfigId.
	#Use the retrieved importConfigId to get CsvImportConfig.		#Use the retrieved importConfigId to get CsvImportConfig.
	#Fetch data from S3 with:<syntaxhighlight lang="javascript">		#Fetch data from S3 with:
			<syntaxhighlight lang="javascript">
	{		{
	Bucket: 'izara-importdata',		Bucket: 'izara-importdata',
Line 39:		Line 45:

	== ProcessFloatingRelationshipsHdrDsq ==		== ProcessFloatingRelationshipsHdrDsq ==

	#Query FloatingRelationships using importBatchId.		#Query FloatingRelationships using importBatchId.
	#Process based on action type (create or update):		#Process based on action type (create or update):
Line 54:		Line 61:
	#Loop to process each PendingLink.		#Loop to process each PendingLink.
	#Run validateLinkConfig to check RequiredOnCreate.		#Run validateLinkConfig to check RequiredOnCreate.
	#If no RequiredOnCreate, create AwaitingMultipleSteps:<syntaxhighlight lang="javascript">		#If no RequiredOnCreate, create AwaitingMultipleSteps:
			<syntaxhighlight lang="javascript">
	await asyncFlowSharedLib.createAwaitingMultipleSteps(		await asyncFlowSharedLib.createAwaitingMultipleSteps(
	_izContext,		_izContext,
Line 91:		Line 99:

	== ProcessPendingObjectHdrDsq ==		== ProcessPendingObjectHdrDsq ==

	#Query PendingObjectMain using importBatchId.		#Query PendingObjectMain using importBatchId.
	#Loop through each PendingObjectMain.		#Loop through each PendingObjectMain.
Line 101:		Line 110:

	== ExternalActionComplete ==		== ExternalActionComplete ==

	#Receive objType, objInstanceFull, errorFound, passBackProperty from ExternalService.		#Receive objType, objInstanceFull, errorFound, passBackProperty from ExternalService.
	#Extract pendingObjectId from passBackProperty, locate AwaitingMultipleSteps, and remove it.		#Extract pendingObjectId from passBackProperty, locate AwaitingMultipleSteps, and remove it.
Line 117:		Line 127:
	#If there are errorFounds → update PendingLink as error; otherwise → mark it complete.		#If there are errorFounds → update PendingLink as error; otherwise → mark it complete.
	#Remove AwaitingMultipleSteps and call clearAllAwaitingSteps.		#Remove AwaitingMultipleSteps and call clearAllAwaitingSteps.


			[[Category:Working documents\| 2025-11-24]]

Jammy at 06:43, 25 November 2025

2025-11-25T06:43:33Z

← Older revision		Revision as of 06:43, 25 November 2025
Line 2:		Line 2:
	== ProcessCsvFile ==		== ProcessCsvFile ==
	#Receive bucketName and fieldName.		#Receive bucketName and fieldName.
	#Define the range, e.g. range = "bytes=0-5000".		#Define the range e.g. range = "bytes=0-5000".
	#Fetch data from S3 with:<syntaxhighlight lang="javascript">		#Fetch data from S3 with:<syntaxhighlight lang="javascript">
	{		{

Jammy: Created page with "= Process ImportData = == ProcessCsvFile == #Receive bucketName and fieldName. #Define the range, e.g. range = "bytes=0-5000". #Fetch data from S3 with: { Bucket: 'izara-importdata', Key: 'csv/xxxx.csv', Range: 'bytes=0-5000' } #Take the importBatchId from the S3 metadata and use it to get ImportBatchMain to retrieve importConfigId. #Use the retrieved importConfigId to get CsvImportConfig. #Use the Cs..."

2025-11-24T10:12:09Z

Created page with "= Process ImportData = == ProcessCsvFile == #Receive bucketName and fieldName. #Define the range, e.g. range = "bytes=0-5000". #Fetch data from S3 with:<syntaxhighlight lang="javascript"> { Bucket: 'izara-importdata', Key: 'csv/xxxx.csv', Range: 'bytes=0-5000' } </syntaxhighlight> #Take the importBatchId from the S3 metadata and use it to get ImportBatchMain to retrieve importConfigId. #Use the retrieved importConfigId to get CsvImportConfig. #Use the Cs..."

New page

= Process ImportData =
== ProcessCsvFile ==
#Receive bucketName and fieldName.
#Define the range, e.g. range = "bytes=0-5000".
#Fetch data from S3 with:<syntaxhighlight lang="javascript">
{
Bucket: 'izara-importdata',
Key: 'csv/xxxx.csv',
Range: 'bytes=0-5000'
}
</syntaxhighlight>
#Take the importBatchId from the S3 metadata and use it to get ImportBatchMain to retrieve importConfigId.
#Use the retrieved importConfigId to get CsvImportConfig.
#Use the CsvImportConfig as the main configuration for processing.
#Loop through the S3 data to extract the titleRow, collecting all field names.
#After extracting all title rows, update fieldNames in ImportBatchMain.
#If S3 data is not fully read, continue with pagination to fetch the next portion.
#When the file ends, send importBatchId to ProcessRowData.

== ProcessRowData ==
#Receive importBatchId.
#Use the received importBatchId to get ImportBatchMain and extract importConfigId.
#Use the retrieved importConfigId to get CsvImportConfig.
#Fetch data from S3 with:<syntaxhighlight lang="javascript">
{
Bucket: 'izara-importdata',
Key: 'csv/xxxx.csv',
Range: 'bytes=0-5000'
}
</syntaxhighlight>
#Loop through the fetched data and store each row into dataRow, including pendingObjects and tempRelationship.
#Validate Schema, Validate Type, Check Required fields, and Check Action (create/update/reference/error).
#Check ImportUserReference to see if this object was already created; if yes, change Action from create → reference.
#Store PendingObjectMain, PendingObjectReference, PendingLink, and FloatingRelationships.
#If the file is not finished, re-invoke itself with the new startByte:
#*If not timed out → continue looping.
#*If timed out → perform pagination.
#If the file is finished, send importBatchId to ProcessFloatingRelationshipsHdrDsq.

== ProcessFloatingRelationshipsHdrDsq ==
#Query FloatingRelationships using importBatchId.
#Process based on action type (create or update):
#*create
#**Create PendingLink for relationships where action = create, and update FloatingStatus: "complete".
#*update
#**Update the existing relationship using data from object A/B.
#**Save AwaitingStep.
#**Call sendMessageToExternalForUpdateRelationship, sending: firstObject, secondObject, relType, relId, relationshipProperties, relationshipDirection.
#If not finished, continue with pagination.
#If fully processed, send importBatchId to ProcessPendingLinkHdrDsq.

== ProcessPendingLink ==
#Query PendingLink using importBatchId.
#Loop to process each PendingLink.
#Run validateLinkConfig to check RequiredOnCreate.
#If no RequiredOnCreate, create AwaitingMultipleSteps:<syntaxhighlight lang="javascript">
await asyncFlowSharedLib.createAwaitingMultipleSteps(
_izContext,
[
asyncFlowSharedLib.createAwaitingStepId(pendingObjectMainOfObjectA.pendingObjectId, consts.PREFIX_CREATE_PENDINGLINK), // object A
asyncFlowSharedLib.createAwaitingStepId(pendingObjectMainOfObjectB.pendingObjectId, consts.PREFIX_CREATE_PENDINGLINK) // object B
],
asyncFlowSharedLib.createPendingStepId(queryPendingLink.pendingLinkId, consts.PREFIX_CREATE_PENDINGLINK) // pendingLink
);
</syntaxhighlight>
#If RequiredOnCreate exists, create two AwaitingMultipleSteps:
#*<syntaxhighlight lang="javascript">
await asyncFlowSharedLib.createAwaitingMultipleSteps(
_izContext,
[asyncFlowSharedLib.createAwaitingStepId(objectRequiredOnCreate.pendingObjectId, consts.PREFIX_IMPORTBATCHMAIN)],
asyncFlowSharedLib.createPendingStepId(importBatchId, consts.PREFIX_IMPORTBATCHMAIN)
);
</syntaxhighlight>
#*<syntaxhighlight lang="javascript">
await asyncFlowSharedLib.createAwaitingMultipleSteps(
_izContext,
[asyncFlowSharedLib.createAwaitingStepId(otherPendingObject.pendingObjectId, consts.PREFIX_CREATE_PENDINGOBJECT)],
asyncFlowSharedLib.createPendingStepId(objectRequiredOnCreate.pendingObjectId, consts.PREFIX_CREATE_PENDINGOBJECT)
);
</syntaxhighlight>
#Update the PendingObjectMain of the RequiredOnCreate object:<syntaxhighlight lang="javascript">
{
"attributeName": "requireOnCreateLinkIds",
"value": [pendingLinkId],
"action": "SET",
"listAppend": true
}
</syntaxhighlight>
#If more data remains, continue pagination.
#If all processed, send importBatchId to ProcessPendingObjectHdrDsq.

== ProcessPendingObjectHdrDsq ==
#Query PendingObjectMain using importBatchId.
#Loop through each PendingObjectMain.
#If the object has RequiredOnCreate, do not send to ExternalService.
#If it has no RequiredOnCreate:
#*Create AwaitingMultipleSteps using PREFIX_IMPORTBATCHMAIN + pendingObjectId.
#*Send to ExternalService according to Action (create, get, update), including:
#**callingFlow = ExternalActionCompleteHdrSqs
#**callingFlowProperties = PREFIX_IMPORTBATCHMAIN + pendingObjectId

== ExternalActionComplete ==
#Receive objType, objInstanceFull, errorFound, passBackProperty from ExternalService.
#Extract pendingObjectId from passBackProperty, locate AwaitingMultipleSteps, and remove it.
#If errorFound, run recursiveErrorPendingObject.
#If no error, update status to "complete".
#Create pendingObjectAwaitingId to find objects that were waiting for this one.
#If there are objects waiting:
#*If they still wait for others → remove their AwaitingMultipleSteps.
#*If not → execute the RequiredOnCreate object and create its relationship.
#Check if all objects are completed. If yes → remove awaitingStep of ImportBatchMain and call clearAllAwaitingSteps.
#Return { returnValue: "externalActionComplete finish" }.

== CreateLinkExternalComplete ==
#Receive firstObject, secondObject, relType, errorFounds, passBackProperties from ExternalService.
#Extract pendingLinkId from passBackProperties, find AwaitingMultipleSteps, and remove it.
#If there are errorFounds → update PendingLink as error; otherwise → mark it complete.
#Remove AwaitingMultipleSteps and call clearAllAwaitingSteps.