2023-03-12 - ImportData CSV Process
Revision as of 12:15, 12 March 2023 by Sven the Barbarian (talk | contribs) (Created page with "Service - Import Data = Process = == Client sends file == # Frontend chooses CsvImportConfig (and can create new), selects file to upload # After clicking submit client requests permission to upload to S3, then sends request to create new importBatchMainId (if can do both in same request is better), then sends file to S3 with indicator of the importBatchMainId, eg as a tag on the file # Can perhaps open a websocket to be notified when complete, or have a page that...")
Process
Client sends file
- Frontend chooses CsvImportConfig (and can create new), selects file to upload
- After clicking submit client requests permission to upload to S3, then sends request to create new importBatchMainId (if can do both in same request is better), then sends file to S3 with indicator of the importBatchMainId, eg as a tag on the file
- Can perhaps open a websocket to be notified when complete, or have a page that polls for status
Process CSV file
- Upload to S3 triggers Lambda in ImportData service to begin processing file
- Read the file in sections for batch processing, maybe as a set section size and handling multiple sections per invocation depending on resource use
- Field order might come from CsvImportConfig or might come from title row, organize these first
- Skip any ignoreRows