Service - Sort Result

Overview

Service that handles sorted results that come from the Service - Search Result Main service. Sorted results are stored/cached so subsequent requests do not need to generate the sorted results again.

Repository

https://bitbucket.org/stb_vit/sortresult_main/src/master/

DynamoDB tables

SortResultMain

• no sort_key

Fields

sortResultId

(partition key)

comes from searchDataId + "_" + sortFieldsHash

searchDataId: {searchType}_{filterId}_{keyPropertiesHash}_{requiredDataHash}

sortFieldsHash: can have multiple sort fields, so we hash them to create the sortFieldsHash unique key

sortFields

array (DynamoDB list) of fields being sorted on and the order (ascending/descending/random)

status

whether sort processing complete for all data yet

initialized | processing | complete

expiry

timestamp when this data expires and needs to be regenerated

SortResultData

Fields

sortResultId

(partition key)

sortValue

(sort key)

value of the field/s being sorted on

for multiple sort fields concatenate the values with "_"

concatenate the records unique id onto the end with "_", otherwise would have duplicates

data

copy of all required data from Search Result Main service

Notes

• The data stored here should include everything the client request might need to render results to the user, so no additional calls are needed (eg all possible media that might be shown as a thumbnail, and how likely that thumbnail would be shown)(eg pricing info min price/max price/most common sold price/etc)
• Client uses sortResultId to pull results
• This service handles pagination of results
• ComplexFilter stores unique ids only for a filter, Service - Search Result Main takes results from ComplexFilter and adds all data client might need. Sort Result Main service copies the data from Search Result Main and structures it for sorted results

Ideas

• date is an interesting sort field, not really used in browse results but would be used a lot in other datasets, it is a filter so could be placed in the complexfilter, but is also something that might change regularly and is based on ordered results, so might be more efficient applied as a sort field in the SortResults, and could filter it there -> my thinking is just treat as a filter and sort separately, could limit changes in filters using date fields so not too many sets of data get generated
• SortResults defines its own expiry date (might not match ComplexFilter/Search Result Main expiry dates), so Search Result Main service must query Sort Result Main to see if have active results, if expired Sort Result Main removes its data, Search Result Main then checks its data/expiry. More fancy: Sort Result Main returns date expired but does not delete data until Search Result Main finishes its tasks, if Search Result Main fails could return Sort Result Main's old data, and/or push back Sort Result Main's expiry date.

References

Efficient sorting DynamoDB structure: https://www.reddit.com/r/aws/comments/7ugukg/dynamodb_not_a_great_option_for_sorting/

Working documents

Working_documents - Sort Result Main