Service - Export Data

Overview

Orchestrates exporting objects/data out of project. Uses SortResult results as base for exported fields and sorting of exported data.

Repository

https://bitbucket.org/izara-core-import-export-data/izara-core-export-data-export-data/src/master/

DynamoDB tables

ExportBatchMain

• One process of exporting data

{
	exportBatchId: "xx", // random uuid
	submittedByUserId: "xx", // submitted by userId
	startTime: currentTime.getTime(),
	exportType: "xx", // "csv"|"xml"|...
	exportConfigId: "yy" // dependent on exportType = "csv"
	overwriteFilters: // object/array(?) of exportConfig filter elements overwriten
	overwriteLinkPath...: // method of overwriting settings for linkPaths, eg which identifier/aggregate to use, eg for finding multiple productAttributes
	exportBatchStatus: "xx", // "processingSortResult" | "processingFeed" | "error" | "complete" 
	userExportConfigOverwritesId: "", // optional
	errorsFound: {}
}

• partition key: exportBatchId
• sort key: {none}

ExportConfig

Base template for creating exportConfig

{
	exportConfigId: "xx", // random uuid
	name: "", // name given by creator
	createdByUserId: "",
	// sortResult params
	objType: {} // base objType of exported data
	sortFields: {} // for SortResult
	requiredDataArray: [] // fields to export, as array so sets column order
	filter/filterArray: {} // see new searchResult filter structure
	// feed params
	recordDeliminator: "\n",
	fieldDeliminator: ",",
	escapeString: "\\",
	removeSpace: true,
	removeTab: true,
	removeTrailingColumns: true,
	enclose: {
		{index of requiredDataArray}: {
			openEnclose: "\"",
			closeEnclose: "\"",
		}
	},
	headerRows: [
		{
			type: "freeText",
			values: ["freetext1", "freetext2"],
		},
		{
			type: "columnHeader", // outputs requiredDataArray fieldNames
		},
	],
	footerRows: [
		// same as preceedingRows
	]
}

• partition key: exportConfigId
• sort key: {none}
• once saved, cannot be edited
• maybe add a suggested overwrite filters property, to make UI easier to work with

UserExportConfigOverwrites

A user save configuration of overwrites they apply to an ExportConfig

{
	userExportConfigOverwritesId: "xx", // random uuid
	name: "", // name given by user
	userId: "", // user who created UserExportConfigOverwrites
	exportConfigId: "xx"
	filterOverwrites: {} // probably point to (nested)filters inside ExportConfig record using array indexes
	overwriteLinkPath...:
}

• partition key: userExportConfigOverwritesId
• can be updated by user
• maybe add list of suggested filters to adjust each time submitted to process (eg date range)

Export Configs

• Users can create ExportConfig, any other user can use, cannot be changed once created (because UserExportConfigOverwrites will point to array indexes in settings)
• UserExportConfigOverwrites is a per user adjustment of an ExportConfig, overwriting some fields, such as a userId filter that specifies which user's data to export
• Other users can create feeds directly from another users UserExportConfigOverwrites
• When a user submits an export request, can do from an ExportConfig and make overwrites, and optionally save the overwrites as an UserExportConfigOverwrites
• ExportBatchMain receives the exportConfigId and overwriteFilters and saves these
• ExportBatchMain optionally receives userExportConfigOverwritesId, for link purposes only, all overwrites are in overwriteFilters param
• this allows user to submit an export request and overwrites that do not match UserExportConfigOverwrites, are used for the single request

Working documents

Working_documents - Export Data