Service - Complex Filter

From Izara Wiki
Jump to navigation Jump to search

Overview

Allows for filters that span nested levels of related entities

Repository

https://bitbucket.org/izara-core-search/izara-core-search-complex-filters

DynamoDB tables

Standard Config Table Per Service

Configuration tags

{
	configKey: "FilterType"
	configTag: "xx" // eg: variant / product / sellOffer
	configValue: {
		childComplexFilterTypes: {
			{child filterType}: {
				"translateIdsServiceName": "xx" // eg: variantManager / sellOfferManager, service than converts child id to parent id
			},
		},
		logicalServiceNames : ["xx", "yy"] // eg: "ProductStandard", "ProductCustom"
	}
}
  • logicalServiceNames should not duplicate over mulitiple FilterType's, because we use this setting to match a Logical Service response to a FilterMain record, need it to uniquely identify one filterMain record
{
	configKey: "ComplexFilterServiceName"
	configTag: "xx" // eg: product / sellOffer
	configValue: "yy" // eg: ComplexFilterProducts
}

{
	configKey: "ExpiryInterval"
	configTag: "ExpiryInterval"
	configValue: xx // minutes(?) complexfilter ttl
}

{
	configKey: "MaxProcessInvocationCount"
	configTag: "MaxProcessInvocationCount"
	configValue: 10 // maximum number of invocations allowed for large operation complex filter processing
}

complexFilter request

{
	filterType: "xx" // eg: "mediaLink",
	type: "group",
	elements:
	[
		{
			type: "complexFilter",
			complexFilter: {
				filterType: "xx", //eg: mediaLinkProperty
				type: "group",
				elements: 
				[
					{
						type: "logical",
						logicalTag: "propertyLabelId", // is used by the logical service/s to decide how to find data

						// the below properties are up to the logicalTag what is needed, but these are common:
						comparison: "equals",
						value: "lksdflkfldfgldfkgjldfg"
					},
				]
			}
		},
		{
			type: "logicalOperator",
			operator: "and"
		},
		{
			type: "complexFilter",
			complexFilter: {
				filterType: "xx", //eg: mediaLinkPropertyTranslation
				type: "group",
				elements: 
				[
					{
						type: "logical",
						logicalTag: "textTag_languageId_text",
						
						//example for matching to specific translation text:
						textTag: "mediaLinkPropertyValue",
						languageId: "en",
						text: "Blue",
						subjectIdentifierPropertyName: "propertyId",
						caseSensitive: true
					},
				]
			}
		},
		...
	]
}

How to request Complex Filter

  • invoke shared library function createFilterMainId which iterates through elements and builds filterMainId by applying operators. There are no database requests in this function so it can be run from any service and can trust the filterMainId will be standardized
  • No cleaning of the filter object is performed, so additional unused properties would result in a different filterMainId, even if the underlying filter is the same
  • calling function saves filterMainId in pending table waiting for filter complete, this needs to be done before calling ProcessComplexFilter to protect against race condition of complex filter completing before pending record saved (probably use AwaitingStep process)
  • All requests to ProcessComplexFilter will send a message out, even if cache exists
  • CallingFlow param allows Complex Filter service to pass a message back to only the calling flow when a request hits cached results

How is filterMainId generated

  • cannot hash entire request elements without iterating through the elements because a final filterMainId of multiple elements is actually the final operation filterMainId (hash of the last element joined with the previous culmination of operation elements)
  • this does not match a hash of the original request object
  • needs to be done this way so we can back track through parent operations in a standardized way when receiving results for each element

Normal Structure ComplexFilter Request

 { //* type: "logical" structure
    objType: {
      serviceTag: "XX",
      objectType: "xx"
    },
    filterMainId: "hash_operation",
    filterElements: { //* this is top filterElements that keep all filterElement of request
      hash_ref_filterElement_A: {
        objType: {
          serviceTag: "XX",
          objectType: "xx"
        },
        filterElement: {
          filterType: "logical",
          partitionKeyValues: {
            xxId: "xx",
            yyId: "yy"
          },
          sortKeyValues: {
            zzId: "zz",
          },
          comparison: "equals",
          conditionalFieldValues: [
            {
              fieldname: "language",
              includeValues: ["th"],
              // excludeValues: ["en","jp"]
            }
          ]
        }
      },
      hash_childComplexFilter: {
        objType: {},
        filterElement: {
           filterType: "childComplexFilter",
           pathLinkType: {
             objType: {
              serviceTag: "YY",
              objectType: "yy"
             },
            relType: {
              serviceTag: "YY",
              relationshipTag: "hasYY"
            },
           direction: "from"
         },
         childFilterElementId: "hash_ref_filterElement_A",
         requestProperties: {
           tag_xxxxx: "xxxxx"
         }
        }
      },
      hash_ref_filterElement_B: {
         //* normalize structure
      },
      hash_operation: {
        objType: {},
        filterElement: {
          filterType: "operation",
          filterElementLeft: "hash_childComplexFilter",
          filterElementRight: "hash_ref_filterElement_B",
          operator: "AND" | "OR"
        }
      }
    }
  }

example pattern each filterType structure in filterElement

[
  {
    filterType: "logical",
      partitionKeyValues: {
            xxId: "xx",
            yyId: "yy"
      },
      sortKeyValues: {
            zzId: "zz",
      },
      comparison: "equals",
      conditionalFieldValues: [
        {
          fieldname: "language",
          includeValues: ["th"],
          // excludeValues: ["en","jp"]
        }
     ]
  },
  { 
    filterType: "childComplexFilter",
    filterElements: { //* use when hash own filterMainId, but when keep in top filterElmenets must delete 
       ref_filterElement_A: {
         //* normalize structure
      }
    },
    pathLinkType: {
       objType: {
         serviceTag: "YY",
         objectType: "yy"
       },
       relType: {
         serviceTag: "YY",
         relationshipTag: "hasYY"
       },
       direction: "from"
    },
    childFilterElementId: "ref_filterElement_A",
    requestProperties: {
       tag_xxxxx: "xxxxx"
    }
  },
  {
    filterType: "operation",
    filterElements: { //* use when hash own filterMainId, but when keep in top filterElmenets must delete 
      ref_filterElement_A: {
        //* normalize structure
      },
      ref_filterElement_B: {
        //* normalize structure
      }
    },
    filterElementLeft: "ref_filterElement_A",
    filterElementRight: "ref_filterElement_B",
    operator: "AND"
  },
  {
    filterType: "translateIds ",
    childIdentifiers: [
      {
        xxId: "111"
      },
      {
        xxId: "222"
      }
    ],
    pathLinkType: {
      objType: {
        serviceTag: "yy",
        objectType: "yy"
      },
      relType: {
        serviceTag: "yy",
        objectType: "hasYY"
      },
      direction: ""
    },
    requesetProperties: {
      tag_quantity: 20
    }
  },
  { //* traversal has hopsPassObjType
    filterType: "traversal",
    traversals: [
      {
        links: [
         {
           relType: {},
           direction: ""
         },
         {
           relType: {},
           direction: ""
         }
        ],
      //* other settings
 	fields: { //* optional
          categoryId: 'sellOfferId_value',
           productId: 'sellOfferId_value'
        },
       relationshipProperties: { timestamp: 'sellOfferId_value' }, //* optional
       hopsStart: 1, //* optional default = 1
       hopsEnd: 10, //* optional
       hopsPassObjType: {
          serviceTag: "",
          objectType: ""
       },
       objType: {
          serviceTag: "",
          objectType: ""
       }
     },
     {
       fields: {
          name: { //fieldName
           anyValue: [ 
              'alice',
              'john',
           //...
          ]
         },
         xxId_fieldname: "xxxx"
       }
     //example other fields pattern: //* front end not yet.
     // fields: {
     //   fieldName: {
     //     comparison: "greaterThan", // greaterThan | lessThan | lessThanOrEqual | greaterThanOrEqual
     //     value: 100
     //   }
     // }

     // fields: {
     //   fieldName: {
     //     comparison: "between",
     //     valueBetweenStart: 100,
     //     valueBetweenEnd: 200
     //   }
     // }
     }
   ]
  },
  { //* traversal no hopsPassObjType    
    filterType: "traversal",
    traversals: [
      {
        links: [
          {
            relType: {},
            direction: ""
          },
          {
            relType: {},
            direction: ""
          }
        ],
        //* other settings
        hopsPassObjType: {
          serviceTag: "",
          objectType: ""
        },
        objType: {
          serviceTag: "",
          objectType: ""
        }
      },
      {
        fields: {
          xxId: "xxxx"
        }
      }
    ]
  }
]

example hash filterElement of each filterType

let hash_ref_filterElement_A = hash({
 objType: {
   serviceTag: "",
   objectType: ""
 },
 filterElement: {
   filterType: "logical" | "childComplexFilter" | "operation" | "translateIds" | "traversal",
   filterElements: { //* use in filterType: "childComplexFilter" | "operation"
     //* all ref filterElement that this filterElement ref
   },
   //* other parameters depend on the filterType.
   //..
   //..
 }
};

Working documents

Complex Filter

Retrieved from "https://izara.io/wiki/index.php?title=Service_-_Complex_Filter&oldid=4182"