2020-11-04 - Sort Result - Functions

Service - Sort Result

Lambda Functions

initSortResult

/**
 * Initial request that checks if a matching, not expired, sort result record exists, creates a new record if not, returns sortResultId
 * @param {string} event.searchDataId - {searchType}_{filterId}_{keyPropertiesHash}_{requiredDataHash}
 * @param {Object[]} event.sortFields - sort fields
 * @param {string} [event.sortFields[].dataField] - name of the field being sorted, not used if randomSeed is set
 * @param {string} [event.sortFields[].randomSeed] - set when wanting random order only
 *
 * @returns {string} sortResultId
 */

logic

1. Create sortResultId by invoking 2020-11-04 - Sort Result - Functions:CreateSortResultId
2. Query SortResultMain table
3. If not have existing record
   1. save record into SortResultMain table
   2. status = new_sortresult_created
   3. if error when creating new record (eg another process already created the record), retry query SortResultMain table and check again
4. If have existing record, expired, and status complete
   1. update SortResultMain record
   2. status = waiting_for_searchresults
   3. conditional update, only if status still complete, if not (eg another process already updated the record), retry query SortResultMain table and check again
5. return sortResultId and status

processSortResult

/**
 * Pulls data from Search Result service and saves into SortResultData table
 * @param {string} event.searchDataId - {searchType}_{filterId}_{keyPropertiesHash}_{requiredDataHash}
 * @param {Object[]} event.sortFields - sort fields
 *
 */

logic

• moving to new logic that saves directly into SortedData from SearchResultData

1. Create sortResultId by invoking 2020-11-04 - Sort Result - Functions:CreateSortResultId
2. Add a retry loop for the below to allow for retrying after conditional statement does not pass
   1. Query SortResultMain table
   2. If status is waiting_for_searchresults
      1. Do a conditional update of SortResultMain record checking status = waiting_for_searchresults, setting status to processing
      2. If conditional statement does not pass retry by "continue"'ing back into retry loop
      3. If conditional statement passes remove all old records from SortResultData
   3. If status is new_sortresult_created
      1. Do a conditional update of SortResultMain record checking status = new_sortresult_created, setting status to processing
      2. If conditional statement does not pass retry by "continue"'ing back into retry loop
   4. If status is processing | error | complete, return
3. .. want to pull data from SearchResultData table, insert into SortResultData table but sorted according to SortFields -- try to work out most effective method for large sets of data
4. Need to copy across all data (eg reqData values)
5. * It might be possible to batch insert Data records into Dynamo to save on resources
6. ..
7. this could be put in a separate Lambda if processing takes too long
8. ..
9. iterate through all StringData/NumericData, inserting records into SortedData table according to their sorted location (increment sortId per record) - put in separate function so easy to split into separate Lambda later
10. .. eg:
11. recursive function that receives one lower level result and queries StringData/NumericData (depending on sortType) for all matching records (records will be sorted in correct order by sortValue/sort key)
12. for each record found:
    1. if is not the final level, invoke recursive function for that value
    2. else:
       1. save each value into SortedData
       2. increment sortId
       3. return new sortId so next records increment correctly
13. ..
14. Conditional update of SortResultMain record checking status = processing
15. status = complete (or error if failed for some reason)
16. expiry = current timestamp + expiryTimeInSec from Config table
17. If conditional statement does not pass should have an error, should not happen

getSortResults

/**
 * Request a page of data
 * @param {string} sortResultId
 * @param {numeric} startId - first sortId to return
 * @param {numeric} limit - number of records to return
 * @param {boolean} event.reverse - true|false, default false, reverse the sort results
 *
 */

handler

• triggered by API gateway request

logic

1. query SortedData table for a set of data, starting from startId, with limit number of records returned

Functions

CreateSortResultId

/**
 * Creates sortResultId from searchDataId and sortFields
 * @param {string} searchDataId
 * @param {Object[]} sortFields
 *
 * @returns {string} sortResultId
 */
module.exports.CreateSortResultId = (searchDataId, sortFields) => {

logic

1. conatenate searchDataId + "_" + hash(sortFields)