Triggered Cache

Overview

System of storing a cached object until a trigger (event) happens. The cached object has a flow that creates the data stored in the cache, this flow takes time, we attempt to have one request re-process the object at any time to avoid corrupt data from race conditions.

TriggerFlow

TriggeredCache has to forms, the basic form is an immediate event happens that resets the cache status, after the status is reset the next request for the cache object will initiate processing.

Second form is the cache has a TriggerFlow, this means another flow is the event that resets the cache, however that flow must complete before the cached data can be used. When the triggerFlow begins the TriggeredCache is reset and must be re-processed, but we do not begin re-processing until the triggerFlow is complete.

To allow this to happen if a request comes in after the triggerFlow has begun but before it has finished we store an awaitingStep that is checked once the flow ends, if found then it begins processing at that point.

If the first flow after resetting comes in after the flow is complete, then processing can begin immediately.

Ensuring one request handles processing

Once we determine that a request will begin processing the triggeredCache we only want one request to handle processing, we do this by using a conditional DynamoDB update (in performCheckConditionUniqueRequestId) checking status has not changed (been set to processing) since we queried the record.

Idempotence

There is the chance the invocation that processes the object may fail before completion, eg if Lambda times out, this would cause the SQS message triggering the lambda to be resent, in this case we want the second request to begin processing so we also test to see if uniqueRequestId matches the current request, if yes we continue processing.

Data Structure

triggeredCache fields can be stored in the objects primary Dynamo record, when we check the triggeredCache we also return the full Dynamo record so it can be used in the resulting logic (eg sent to a complete topic if using cache).

One object might have multiple triggeredCache sets attached to it, eg if there are multiple flows or tasks that need to be performed for that object. All sets can be saved in the main Dynamo record together, use triggeredCache prefix to separate the fields.