Service - Export Data - Revision history

Sven the Barbarian at 21:37, 2 May 2024

2024-05-02T21:37:51Z

← Older revision		Revision as of 21:37, 2 May 2024
Line 48:		Line 48:
	recordDeliminator: "\n",		recordDeliminator: "\n",
	fieldDeliminator: ",",		fieldDeliminator: ",",
			escapeString: "\\",
			removeSpace: true,
			removeTab: true,
			removeTrailingColumns: true,
	enclose: {		enclose: {
	{index of requiredDataArray}: {		{index of requiredDataArray}: {
Line 54:		Line 58:
	}		}
	},		},
	~~preceedingRows~~: [		headerRows: [
	{		{
	type: "freeText",		type: "freeText",
	~~columns: {~~		values: ["freetext1", "freetext2"],
	~~{index of requiredDataArray}~~: "~~xxx~~" ~~// any text~~
	}
	},		},
	{		{
	type: "~~titleRow~~", // outputs requiredDataArray fieldNames		type: "columnHeader", // outputs requiredDataArray fieldNames
	},		},
	],		],
	~~succeedingRows~~: [		footerRows: [
	// same as preceedingRows		// same as preceedingRows
	]		]

Sven the Barbarian at 21:33, 2 May 2024

2024-05-02T21:33:41Z

@@ Line 75: / Line 75: @@
 * once saved, cannot be edited
 * maybe add a suggested overwrite filters property, to make UI easier to work with
 == UserExportConfigOverwrites ==
@@ Line 129: / Line 94: @@
 * can be updated by user
 * maybe add list of suggested filters to adjust each time submitted to process (eg date range)
 = Export Configs =

Sven the Barbarian at 11:44, 30 April 2024

2024-04-30T11:44:58Z

← Older revision		Revision as of 11:44, 30 April 2024
Line 21:		Line 21:
	exportConfigId: "yy" // dependent on exportType = "csv"		exportConfigId: "yy" // dependent on exportType = "csv"
	overwriteFilters: // object/array(?) of exportConfig filter elements overwriten		overwriteFilters: // object/array(?) of exportConfig filter elements overwriten
			overwriteLinkPath...: // method of overwriting settings for linkPaths, eg which identifier/aggregate to use, eg for finding multiple productAttributes
	exportBatchStatus: "xx", // "processingSortResult" \| "processingFeed" \| "error" \| "complete"		exportBatchStatus: "xx", // "processingSortResult" \| "processingFeed" \| "error" \| "complete"
	userExportConfigOverwritesId: "", // optional		userExportConfigOverwritesId: "", // optional
Line 121:		Line 122:
	exportConfigId: "xx"		exportConfigId: "xx"
	filterOverwrites: {} // probably point to (nested)filters inside ExportConfig record using array indexes		filterOverwrites: {} // probably point to (nested)filters inside ExportConfig record using array indexes
			overwriteLinkPath...:
	}		}
	</syntaxhighlight>		</syntaxhighlight>

Sven the Barbarian at 11:41, 30 April 2024

2024-04-30T11:41:19Z

@@ Line 39: / Line 39: @@
 	name: "", // name given by creator
 	createdByUserId: "",
 	objType: {} // base objType of exported data
 	sortFields: {} // for SortResult
 	requiredDataArray: [] // fields to export, as array so sets column order
 	filter/filterArray: {} // see new searchResult filter structure
+}
 </syntaxhighlight>

Sven the Barbarian at 11:33, 30 April 2024

2024-04-30T11:33:55Z

← Older revision		Revision as of 11:33, 30 April 2024
Line 22:		Line 22:
	overwriteFilters: // object/array(?) of exportConfig filter elements overwriten		overwriteFilters: // object/array(?) of exportConfig filter elements overwriten
	exportBatchStatus: "xx", // "processingSortResult" \| "processingFeed" \| "error" \| "complete"		exportBatchStatus: "xx", // "processingSortResult" \| "processingFeed" \| "error" \| "complete"
			userExportConfigOverwritesId: "", // optional
	errorsFound: {}		errorsFound: {}
	}		}
Line 122:		Line 123:
	* UserExportConfigOverwrites is a per user adjustment of an ExportConfig, overwriting some fields, such as a userId filter that specifies which user's data to export		* UserExportConfigOverwrites is a per user adjustment of an ExportConfig, overwriting some fields, such as a userId filter that specifies which user's data to export
	* Other users can create feeds directly from another users UserExportConfigOverwrites		* Other users can create feeds directly from another users UserExportConfigOverwrites
			* When a user submits an export request, can do from an ExportConfig and make overwrites, and optionally save the overwrites as an UserExportConfigOverwrites
			* ExportBatchMain receives the exportConfigId and overwriteFilters and saves these
			* ExportBatchMain optionally receives userExportConfigOverwritesId, for link purposes only, all overwrites are in overwriteFilters param
			* this allows user to submit an export request and overwrites that do not match UserExportConfigOverwrites, are used for the single request

	= Working documents =		= Working documents =

Sven the Barbarian: Created page with "= Overview = Orchestrates exporting objects/data out of project. Uses SortResult results as base for exported fields and sorting of exported data. = Repository = https://bitbucket.org/izara-core-import-export-data/izara-core-export-data-export-data/src/master/ = DynamoDB tables = == ExportBatchMain == * One process of exporting data { exportBatchId: "xx", // random uuid submittedByUserId: "xx", // submitted by userId startTim..."

2024-04-30T11:27:40Z

Created page with "= Overview = Orchestrates exporting objects/data out of project. Uses SortResult results as base for exported fields and sorting of exported data. = Repository = https://bitbucket.org/izara-core-import-export-data/izara-core-export-data-export-data/src/master/ = DynamoDB tables = == ExportBatchMain == * One process of exporting data <syntaxhighlight lang="JavaScript"> { exportBatchId: "xx", // random uuid submittedByUserId: "xx", // submitted by userId startTim..."

New page

= Overview =

Orchestrates exporting objects/data out of project. Uses SortResult results as base for exported fields and sorting of exported data.

= Repository =

https://bitbucket.org/izara-core-import-export-data/izara-core-export-data-export-data/src/master/

= DynamoDB tables =

== ExportBatchMain ==

* One process of exporting data

<syntaxhighlight lang="JavaScript">
{
exportBatchId: "xx", // random uuid
submittedByUserId: "xx", // submitted by userId
startTime: currentTime.getTime(),
exportType: "xx", // "csv"|"xml"|...
exportConfigId: "yy" // dependent on exportType = "csv"
overwriteFilters: // object/array(?) of exportConfig filter elements overwriten
exportBatchStatus: "xx", // "processingSortResult" | "processingFeed" | "error" | "complete"
errorsFound: {}
}
</syntaxhighlight>

* partition key: exportBatchId
* sort key: {none}

== ExportConfig ==

Base template for creating exportConfig

<syntaxhighlight lang="JavaScript">
{
exportConfigId: "xx", // random uuid
name: "", // name given by creator
createdByUserId: "",
objType: {} // base objType of exported data
sortFields: {} // for SortResult
requiredDataArray: [] // fields to export, as array so sets column order
filter/filterArray: {} // see new searchResult filter structure
}
</syntaxhighlight>

* partition key: exportConfigId
* sort key: {none}
* once saved, cannot be edited
* maybe add a suggested overwrite filters property, to make UI easier to work with

== ExportConfig ==

Base template for creating exportConfig

<syntaxhighlight lang="JavaScript">
{
exportConfigId: "xx", // random uuid
name: "", // name given by creator
userId: "", // user who created ExportConfig
objType: {} // base objType of exported data
sortFields: {} // for SortResult
requiredDataArray: [] // fields to export, as array so sets column order
filter/filterArray: {} // see new searchResult filter structure
}
</syntaxhighlight>

* partition key: exportConfigId
* sort key: {none}
* once saved, cannot be edited
* maybe add a suggested overwrite filters property, to make UI easier to work with

== UsersExportConfig ==

* for efficient querying

<syntaxhighlight lang="JavaScript">
{
userId: "xx", // user who owns the csvImportConfig
exportConfigId: "xx",
}
</syntaxhighlight>

* partition key: userId
* sort key: exportConfigId

== UserExportConfigOverwrites ==

A user save configuration of overwrites they apply to an ExportConfig

<syntaxhighlight lang="JavaScript">
{
userExportConfigOverwritesId: "xx", // random uuid
name: "", // name given by user
userId: "", // user who created UserExportConfigOverwrites
exportConfigId: "xx"
filterOverwrites: {} // probably point to (nested)filters inside ExportConfig record using array indexes
}
</syntaxhighlight>

* partition key: userExportConfigOverwritesId
* can be updated by user
* maybe add list of suggested filters to adjust each time submitted to process (eg date range)

== UsersUserExportConfigOverwrites ==

* for efficient querying

<syntaxhighlight lang="JavaScript">
{
userId: "xx", // user who owns the csvImportConfig
userExportConfigOverwritesId: "xx",
}
</syntaxhighlight>

* partition key: userId
* sort key: userExportConfigOverwritesId

= Export Configs =

* Users can create ExportConfig, any other user can use, cannot be changed once created (because UserExportConfigOverwrites will point to array indexes in settings)
* UserExportConfigOverwrites is a per user adjustment of an ExportConfig, overwriting some fields, such as a userId filter that specifies which user's data to export
* Other users can create feeds directly from another users UserExportConfigOverwrites

= Working documents =

[[:Category:Working_documents - Export Data|Working_documents - Export Data]]

[[Category:Backend services| Export Data]]