2020-10-12 - Search Result add hooks

Service - Search Result Main has some tasks which must be performed by Service - Search Result (handlers), for example DigData, whose requests are hardcoded into the SearchResultMain code.

Some searchTypes will have optional tasks at points within the flow, for example variantProduct will have an extra step after the child product searchType has completed, but before processing DigData for each variant data.

variantProduct handler will save to a working table (StandAloneProducts) all product ids found, as it goes through each variant it removes matching products from this table, the remaining product ids will be stand alone products.

To avoid race conditions we need to save all found product ids to the StandAloneProducts before we start processing DigData on the variants. We have no method to do this at the moment because it has to be done in the searchType's handler service (because is non-standard) and there is no flow to hold back DigData processing once a search result main record is ready to start processing.

variantProduct handler also needs to copy StandAloneProducts to copy into variantProduct's Data table, after all variant data is finished and before processing any parents or completing the searchResultMain record.

Create hook

We can add hooks to a searchType's config that control whether a searchType requires a function to be invoked before processing DigData requests.

We want these hooks to use serverless flow, no blocking, so need to send message out, and have queue that receives response. SearchResultMain can send an SQS message directly to handler service's hookBeforeDigData's trigger queue. When hookBeforeDigData is finished it can send an SQS message directly to SearchResultMain's completeHookBeforeDigData queue

Was considering allowing multiple hooks but that would require more code and config as we could not standardize the name of the function being invoked, instead have only one hook, true or false, and it is up to the searchType's handler service if it wants to spread it out to more services.

hookBeforeDigData

Hook before sending messages to DigData in ProcessFindRequiredData, call it hookBeforeDigData, if this is set in searchType's config then we try to invoke lambda function in handler service named hookBeforeDigData

If there is no hookBeforeDigData we can process ProcessFindRequiredData/DigData immediately, so move that code to a new function that can also be called by SearchResultMain's completeHookBeforeDigData function

hookAfterDigData

Hook after all data completed, before checking for parents and setting searchResultMain to complete.

Move code in completeSearchResultMain to a shared library function. Change to checking searchType's config for hookAfterDigData, if not set can invoke shared lib, if it is set send SQS to handler service's hookAfterDigData function, if that function completes send SQS to SearchResultMain's completeHookAfterDigData function

idea (rejected): Add per SearchResultMain config setting

- config setting will check if task done yet - when process 1 data record (eg 1 variant) in search result handler's DigData function, check if config value set yet, if not then conditional update it and process the work - Good: no new serverless flow needed - Unacceptable: we could control it so only one DigData process does the additional work needed, but there is the chance of a race condition if a second DigData request wants to remove a product entry from the StandAloneProducts before that product has been added there