Service - Import Data
Overview
Orchestrates importing of objects/data into project.
Repository
https://bitbucket.org/izara-core-import-export-data/izara-core-import-data-import-data/src/master/
DynamoDB tables
Standard Config Table Per Service
Configuration tags
{
configKey: "objectType",
configTag: "xx" // {objectType, eg: sellOffer/Product/VariantProduct etc..}
configValue: {
createObjectServiceName: "xx" // {service name service that handles this type}
parentLinks: {
{parent objectType}: {
{linkTag}: {
"linkType": "yy", // Dependent|Independent
"separateDependentLinkCreate": true, // for Dependent only, default is false
"createLinkServiceNames": "yy", // if not exist, does not send to external service, sets link as complete
}
}
},
childObjectTypes: [] // maybe not needed?
fieldNames: { // list of possible fieldNames, will ignore fields found in file that are not listed here
{fieldName}: {
// no settings yet
}
}
}
},
- separateDependentLinkCreate: if set to true then a dependent link will send a separate request to external service after parent and child objects created. Default is false where the link is created in the same request as creating the child object.
ImportBatchMain
{
importBatchId: "xx", // random uuid
userId: "xx", // target userId
submittedByUserId: "xx", // submitted by userId
startTime: currentTime.getTime(),
importType: "xx", // "csv"|"xml"|...
importConfigId: "yy" // dependent on importType = "csv"
importBatchStatus: "xx", // "* NOT YET:processingRawRecords" | "processingObjects" | "error" | "complete"
errorsFound: {},
processingError: false // true|false, if processing object/links has error set this to true
}
- partition key: importBatchId
- sort key: {none}
PendingObjectMain
One item per object that needs to be created.
{
importBatchId: "xx",
pendingObjectId: "xx", // hash of object with importBatchId, userId, objectType, and fields properties
objectType: "xx", // eg variant|product|sellOffer|sellOfferPrice|sellOfferPlan|...
fields: {}, // key is the name of the field
identifierIds: { // used to identify an existing object, fields is ignored if this is found
{name of identifier property}: {value}
},
action: "xx", // create|update|reference|error
rowNumber: "xx", // the order in which this record was extracted from source file
objectMainStatus: "xx", // processing|creating|complete|error
errorsFound: {},
}
- partition key: importBatchId
- sort key: pendingObjectId
- there is the possibility of a feed sending the same objectType and fields multiple times in a single request, above design will only handle this once, I think this is OK, maybe do a check and add error if duplicates found
PendingObjectReference
Creates a link between submitted referenceId and saved object, so can find when other objects reference it.
{
importBatchId: "xx",
referenceId: "xx", // {feed supplied referenceId}
pendingObjectId: "xx",
}
- partition key: importBatchId
- sort key: referenceId
- when creating maybe throw error if item exists with different pendingObjectId
PendingLink
One item per link between objects.
{
importBatchId: "xx",
pendingLinkId: "xx", // {pendingObjectId}_{referenceId}_{relationshipTag}
linkStatus: "xx", // processing|creating|complete|error
errorsFound: {}, // previously not added, considering adding so links can store their errors independent of pendingObjects
}
- partition key: importBatchId
- sort key: pendingLinkId
CsvImportConfig
{
csvImportConfigId: "xx", // random uuid
userId: "xx", // user who controls/created/owns the config
recordDeliminator: "\n",
fieldDeliminator: ",",
escapeString: "\\",
removeFloatingEscapeString: true, // default: false, removes single escape strings that do not precede expected escapedStrings
removeWhiteSpace: false, // default: true, removes any spaces/tabs/enters at start or end of fields
fieldNames:{
fixed: { // used to fix which columns are which fieldNames, eg when no title row exists
columnNumber: {
objectTypeConfigIndex: 0,
fieldname: 'name', // optional, if not set use standard method in objectTypeConfigIndex
objType: { // optional, if not set use standard method in objectTypeConfigIndex
serviceTag: "xxx",
objectType: "xxx",
},
instance: 'xx' , // optional, if not set use standard method in objectTypeConfigIndex
openEnclose: "\"", // optional , if not set use titleRowOpenEnclose
closeEnclose: "\"" // optional , if not set use titleRowCloseEnclose
}
}
// or
titleRow: # // which row has the fieldNames
titleRowOpenEnclose: "\"",
titleRowCloseEnclose: "\"",
replacefieldNames: {
{fromFieldName}: "{toFieldName}", // refactor existing fieldNames to our names
},
},
ignoreRows: [], // row numbers to skip
overwriteColumnName: {
"columnName": "{replaceToValue}", // completely change columnName before extractiing objectType, instance, fieldname
},
objectTypes: [
{
setObjectTypeFieldNames: { // if a value in a specific field sets the rows objectType
{fieldName}: {
{fieldValue}: {
serviceTag: "xx",
objectType: "yy"
} // index is the value found in the field, matches to the specified objectType
},// .. can look in multiple fields to find the matching objectType, will use the first one found
},
objType: {
serviceTag: "xxx",
objectType: "xxx",
}
searchPattern: "xxx", // regexp search of the column name, if matches then is the associated objectType
instancePattern: "after productattribute and before colon", // extract from the column name the instance identifier for this object, eg if one row creates multiple product attributes. Optional, if not set use empty string as instance
fieldNamePatterns: [ // optional, if not set will check fieldNameSearchPattern, or if none found/set will be a null column
{
fieldNamePattern: "yyy", // extract from the column name the fieldname for this object
fieldName: "zzzz",
},
// ....
],
// check fieldNamePatterns first, if none match, check fieldNameSearchPattern to extract the fieldname
fieldNameSearchPattern: "after colon", // regExp that pulls out the fieldname, optional
referenceFieldNames: ["xx","yy"], // columnNames that set the string referenceId for each mainObjectType pendingObject, array in case multiple fields might set reference, if multiple are set is not defined which will be used
referenceLinks: {
linkTargetXXX: {
relType: {
serviceTag: "xxx",
relationshipTag: "yyy",
},
direction: "incoming" // incoming or outgoing
},
...
},
// referenceLinks or automaticLinks
automaticLinks: [ // automatically create links between objects created on the same record
{
objType: {// which objectType to link to
serviceTag: "xxx",
objectType: "xxx",
},
instance: "tt", // which instance identifier to link to
relType: {
"serviceTag": "xxx",
"relationshipTag": "yyy",
},
direction: "incoming" // incoming or outgoing
}
// ..
],
actionField: {
fieldName: "xx",
createValue: "c",
updateValue: "u",
referenceValue: "r"
},
versionDataIds: [
{
versionedDataLabel: "rateTableRates",
fieldName: "xxxx"
},
{
versionedDataLabel: "rateTableRates2",
fieldName: "zzzzz"
}, ...
],
enclose: [
{
openEnclose: "\"",
closeEnclose: "\"",
alwaysEnclose: "always" // "always"|"optional", default always // NOT SURE NEEDED, maybe always check if exists or not
fieldNames: [],
}
],
overwriteValue:{
{fieldName}: {
{value}: {overwriteValue},
eg:{
off:"false", // can be string or Boolean
n:false,
0:false
}
}
}
},
// ...
],
floatingRelationships:[
{
setRelationshipTagFieldNames: { // if a value in a specific field sets the relationshipTag
{fieldName}: { // fieldValue is the value found in the field, value is the relationshipTag
{fieldValue}: {
relType: {
"serviceTag": "xxx",
"relationshipTag": "yyy",
},
direction: "incoming" // incoming or outgoing
},
....
},
// .. can look in multiple fields to find the matching objectType, will use the first one found
},
relationships:{ // relationshipTag: "hasRateTable"
relType: {
serviceTag: "xxx",
relationshipTag: "yyy",
},
direction: "incoming" // incoming or outgoing
},
searchPattern: "xxx", // regexp search of the column name, if matches then is the associated relationshipTag
instancePattern: "after hasRateTable and before colon", // extract from the column name the instance identifier for this relationshipTag, eg if one row creates multiple 'has' relationships. Optional, if not set use empty string as instance
relationshipPropertyPatterns: [ // optional, if not set will check relationshipPropertySearchPattern, or if none found/set will be a null column
{
relationshipPropertyPattern: "yyy", // extract from the column name the fieldname for this object
relationshipProperty: "zzzz",
},
// ....
],
// check relationshipPropertyPatterns first, if none match, check relationshipPropertySearchPattern to extract the relationshipProperty name
relationshipPropertySearchPattern: "after colon", // regExp that pulls out the relationshipProperty, optional
objectBReferenceLinks: [
"reateTableRef01"
],
objectAReferenceLinks: [
"delMethodStd"
],
enclose: [
{
openEnclose: "\"",
closeEnclose: "\"",
alwaysEnclose: "always" // "always"|"optional", default always // NOT SURE NEEDED, maybe always check if exists or not
fieldNames: [],
}
],
},
// ...
]
}
- partition key: csvImportConfigId
- sort key: {none}
actionColumn
Is optional, controls whether this pendingObject will be created, updated, or used as a reference for links. If actionColumn is not set then system will perform the action according to the below rules:
- Create: if no identifier fields are not set then will attempt to create the object according to found fields
- Update: if all identifier fields are set and some other fields are set, will attempt to update
- Reference: if all identifier fields are set and no fields are set will use as a reference (check exists)
- case: some identifiers are set: process pendingObject set as error/failed
If actionColumn is set will fail if the following:
- Update or Reference and not all identifier fields set
- Create and any identifier fields set
- value does not match any of the create/update/reference values
If actionColumn is set to Reference, any fields found will be ignored.
UsersCsvImportConfig
{
userId: "xx", // user who owns the csvImportConfig
csvImportConfigId: "xx",
}
- partition key: userId
- sort key: csvImportConfigId
FloatingRelationships
{
importBatchId: "xx", // random uuid
identifierRelationshipsId: "xx", // random uuid
referenceProperty:{
objectAReferenceLinks: "rateTableRef01",
objectBReferenceLinks: "delMethodStd01",
relationshipTag: "hasRateTable",
relationshipProperty: {...}
}
}
- partition key: importBatchId
- sort key: identifierRelationshipsId
External service requests
Both createObject and createLink.
- standard Lambda like ProcessLogical and FindData
- ImportData subscribes to standard complete topic: CreateObjectComplete
- Each external service can handle processing and recording the pendingObjectId to return in their own way, eg by having another table that links the external service's identifier with the ImportData pendingObjectId, querying this table to make CreateObjectComplete message
- userId is also sent as need to record user creating objects
- pendingObjectId is also sent, must be returned in CreateObjectComplete message
Linking objects
Two types of linking:
- Independent: Objects can be created independently of each other in any order
- Dependent: One object must be created before the other in a specific order
for Config setting objectType.parentLinks either object can be the parent for Independent links, for Dependent links the object created first is considered the parent.
Two object types may have multiple types of links connecting them so each parentLinks element has a list of linkTags which reference what type of link is being created.
For each parentLinks element there will be a matching entry in the other object's childObjectTypes array.
One object might have multiple other objects dependent on it to be created, or be dependent on many other objects being created.
Object hierarchy and field schema
- Some fields will be required, some optional
- some fields possibly have system defaults
- perhaps user can setup default templates (do later if has value)
- schema will need to state identifier fields for each object, if set in feed Import Data knows is pointing to existing child/parent object, if empty needs to create new
- perhaps each objectType states it's child objects, as more likely to be aware of these than parent objects
where to store/set schema
Considering external service delivers this to ImportData in Initial Setup, as seed data injected directly into Import Data Config Dynamo table.
Errors during processing
- Each object/link records it's own errorsFound and status
Before sending any request to external services
- up to ProcessPendingLinks
- ImportBatchMain records all errors found, including 1 error per object/linkwith error/s
- Before starting to send requests to external services, if ImportBatchMain has any errors, stop processing
- Have a limit, when adding errors to ImportBatchMain if total errors found exceeds limit, stop processing
After start sending requests to external services
- after ProcessPendingLinks (ProcessPendingObjects)
- processing has begin so we continue until finished, however any object/link sets any remaining connected object/links to error, removing the awaitingSteps
- any object/link error set ImportBatchMain processingError to true, do not store the object/link's errors into ImportBatchMain
- when ImportBatchMain has no more work to do check processingError, if true add an error that some object/link have error