Service - Sort Result

Overview

Service that handles sorted results that come from the Service - Search Result service. Sorted results are stored/cached so subsequent requests do not need to generate the sorted results again.

Repository

https://bitbucket.org/stb_vit/sortresult_main/src/master/

DynamoDB tables

Standard Config Table Per Service

Configuration tags

{
	configKey: "SearchType",
	configTag: "xx" // {eg: sellOffer/Product/VariantProduct etc..}
	configValue: {
		searchResultServiceName: "xx" // {service name of searchResult service that handles this type}
	}
},

SortResultMain

{
	sortResultId: "xx", // main element for one set of search results, comes from {searchType}_{filterMainId}
	requiredData: {}, // same as request
	sortFields: [], // same as request
	requestProperties: {}, // same as request
	createTime: currentTime.getTime(),
	expiryTime: expiryTime.getTime(),
}

• partition key: sortResultId
• sort key: {none}
• sortResultId: searchId + "_" + sortFieldsHash
• sortFieldsHash: Strip out any sortFields elements after a random element is found, as they are unused. can have multiple sort fields, so we hash them to create the sortFieldsHash unique key

SortResultData

{
	sortResultId: "xx", 
	sortId: ##, // numeric increment for sorting
	dataId: "xx", unique id for the data record, from Search Result service
	data: {}, // all required data from Search Result service
}

• partition key: sortResultId
• sort key: sortId

TempStringSort

{
	sortResultIdLevel: "xx", 
	valueDataId: "xx",
	dataId: "xx"
}

• partition key: sortResultIdLevel
• sort key: valueDataId
• sortResultIdLevel: is {sortResultId}_{sortLevel}_{previous sort levels order}
• valueDataId: is {value being sorted}~_~{dataId}
• using tilde in valueDataId should ensure sorting does not mix with dataId, because tilde is sorted last by dynamodb

TempNumericSort

{
	sortResultIdLevel: "xx", 
	number: "xx"
}

• partition key: sortResultIdLevel
• sort key: number
• sortResultIdLevel: is {sortResultId}_{sortLevel}_{previous sort levels order}
• number: is the value of the field being ordered by

TempNumericDataId

{
	sortResultIdLevelNumber: "xx", 
	dataId: "xx"
}

• partition key: sortResultIdLevelNumber
• sort key: dataId
• sortResultIdLevelNumber: is {sortResultIdLevel}_{number value}
• links multiple dataIds with the same numeric value to their sorted record in TempNumericSort

Notes

• The data stored here should include everything the client request might need to render results to the user, so no additional calls are needed (eg all possible media that might be shown as a thumbnail, and how likely that thumbnail would be shown)(eg pricing info min price/max price/most common sold price/etc)
• Client uses sortResultId to pull results, can also request ascending or descending
• This service handles pagination of results
• ComplexFilter stores unique ids only for a filter, Service - Search Result takes results from ComplexFilter and adds all data client might need. Sort Result service copies the data from Search Result and structures it for sorted results

Ideas

• date is an interesting sort field, not really used in browse results but would be used a lot in other datasets, it is a filter so could be placed in the complexfilter, but is also something that might change regularly and is based on ordered results, so might be more efficient applied as a sort field in the SortResults, and could filter it there -> my thinking is just treat as a filter and sort separately, could limit changes in filters using date fields so not too many sets of data get generated, eg by fixing requests to set day timestamps.
• There is some large scale put into DynamoDB commands in this service, might be improved using BatchWriteItem, or improving async code.
• SortResults defines its own expiry date (might not match ComplexFilter/Search Result expiry dates), so Search Result service must query Sort Result to see if have active results, if expired Sort Result removes its data, Search Result then checks its data/expiry. More fancy: Sort Result returns date expired but does not delete data until Search Result finishes its tasks, if Search Result fails could return Sort Result's old data, and/or push back Sort Result's expiry date.

See also

• https://www.reddit.com/r/aws/comments/7ugukg/dynamodb_not_a_great_option_for_sorting/ - Efficient sorting DynamoDB structure

Working documents

Working_documents - Sort Result