Service - Integration Testing

Overview

Service that pulls integration tests from the Integration Test Config service, invokes each test and monitors the steps execute as expected.

This service is intended to be run in a deployed AWS environment, all services and resources required for the integration tests need to be operational.

DynamoDB tables

TestRecord

Fields

integrationTestTag

(partition key)

testStartedTimestamp

(sort key)

time the integration test was started/created.

testCompleteTimestamp

time the integration test completed.

testSettings

straight copy of test configuration testSettings property

stages

an object that holds the details and status of each stage #testRecord.stages structure

testStatus

current status of the integration test, either processing, passed, or failed

testErrors

an array of any misc errors found (probably stored as a set, ordering is not important and should not have any duplicates)

Lambda Functions

initiateIntegrationTest

/**
 * Starts integration test/s
 * @param {string} [event.serviceName] - Only initiate tests where initialStage serviceName matches
 * @param {string} [event.resourceType] - Only initiate tests where initialStage resourceType matches
 * @param {string} [event.resourceName] - Only initiate tests where initialStage resourceName matches
 * @param {string[]} [event.integrationTestTags] - Only initiate tests with matching integrationTestTags
 *
 * @returns {boolean} true if the test was initiated successfully, false if the test could not be initiated (? or an error thrown ?)
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Invoke Service - Integration Test Config:getIntegrationTests to get test configurations of all matching tests
2. For each integration test:
   1. For each integration test stage:
      1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the test stage's config as new properties inputEventConfig and outputEventConfig
      2. For each invokes element:
         1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the stage invoke's config as new properties inputEventConfig and outputEventConfig

            (we could cache getEventConfig results, as many will be duplicated)

      3. Store the config for the initialStage in a variable to be used later in logic

         (we do not want to invoke the initial request until all stages are saved into TestRecord table, to avoid race conditions)

      4. if an initialStage has already been found add an error to testRecord.testErrors and set status testRecord.testStatus to failed
   2. If no initialStage was found add an error to testRecord.testErrors and set status testRecord.testStatus to failed
   3. Save the TestRecord record, including the updated testSettings, testErrors (if any), and testStatus (if not failed, should be processing)
   4. For the initialStage:
      1. Build initial event to start the test using the initialStage's input event
      2. Add intTest-xx correlation ids #Initiating tests
      3. Invoke the initialStage's Lambda function

receiveMsgOutput

/**
 * Triggered when a resource invoked by an integration test returns
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match resource invocation, if yes perform tests on return value and record results
 * @param {object} event.requestParams - The event body received by the resource
 * @param {string} event.serviceName
 * @param {string} event.resourceType
 * @param {string} event.resourceName
 * @param {object} event.returnValue //how to handle errors? Maybe place the actual value into a property, and have other properties for error/other settings
*/
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
2. Find matching stage using #findMatchingStage
3. If findMatchingStage returns false, return
4. For each testRecord.stages.{stageKey}.outputEventTag.properties where testValueMatches != false test if value matches in returnValue
5. ... (to update:) or straight eventValue check? ....
6. ... (to update) or errors? ....
7. Update testRecord.stages.{stageKey}.stageResults.outputResult values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.outputResult does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgResourceOutput might get delivered multiple times, if experience that maybe adjust logic to not record error?)
8. Check if any tests for this stage remain using #checkStageTestsComplete

receiveMsgInvoke

/**
 * Triggered when a resource invoked by an integration test invokes another resource
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match resource invocation, and has an invoke setting that matches message, if yes perform tests on invoke's values and record results
 * @param {object} event.requestParams - The original event body received by the resource
 * @param {string} event.serviceName
 * @param {string} event.resourceType
 * @param {string} event.resourceName
 * @param {object} event.invokeRequestParams
 * @param {string} event.invokeServiceName
 * @param {string} event.invokeResourceType
 * @param {string} event.invokeResourceName
*/
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
2. Find matching stage using #findMatchingStage
3. If findMatchingStage returns false, return
4. Find matching invoke for stage using #findMatchingStageShared, inputting testRecord.stages.{stageKey}.stageConfig.invokes and invoke.. params
5. If no matching invoke found and testRecord.testSettings.errorIfInvokeUndefined set to true:
   1. Add error to testRecord.stages.{stageKey}.stageErrors
   2. Save testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}, resultStatus set to failed
   3. Check if any stageResults for this stage remain using #checkStageTestsComplete
   4. Return false
6. If more than one matching invoke found:
   1. Add error to testRecord.stages.{stageKey}.stageErrors
   2. For each invoke found:
      1. Save testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}, resultStatus set to failed
   3. Check if any stageResults for this stage remain using #checkStageTestsComplete
   4. Return false
7. For each testRecord.stages.{stageKey}.stageConfig.invokes.{invokeKey}.inputEventTag.properties where testValueMatches != false test if value matches in invokeRequestParams
8. ... (to update) or straight value check ....
9. ... (to update) or errors? ....
10. Update testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag} values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag} does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgResourceOutput might get delivered multiple times, if experience that maybe adjust logic to not record error?)
11. Check if any tests for this stage remain using #checkStageTestsComplete

Functions

findMatchingStage

/**
 * Checks to see if received message has a matching stage
 * @param {object} requestParams - The event body received by the resource
 * @param {string} serviceName
 * @param {string} resourceType
 * @param {string} resourceName
 * @param {object} testRecord
 * @param {string} resultTag
 *
 * @returns {number} key of found stage or null if none found
 */
module.exports.findMatchingStage = (stage, invokes) => {

logic

1. find matching stage/s using #findMatchingStageShared
2. If no matching stage found and testRecord.testSettings.errorIfStageUndefined set to true:
   1. update testRecord database record, adding an element to testRecord.testErrors (can use UpdateItem.Add method)
   2. Return false
3. If more than one matching stage found, for each stage found:
   1. add error to testRecord.stages.{stageKey}.stageErrors property
   2. save testRecord.stages.{stageKey}.stageResults.[resultTag], resultStatus set to failed (probably skip saving additional fields like requestParams and returnValue, is an edge case)(don't worry about race conditions, should not cause serious issues)
   3. Check if any stageResults for this stage remain using #checkStageTestsComplete
   4. Return false
4. no errors can return element key of found stage

findMatchingStageShared

/**
 * Checks to see if received message has a matching stage
 * @param {string} serviceName
 * @param {string} resourceType
 * @param {string} resourceName
 * @param {object} stages - either testRecord.stages array or testRecord.stages.{stageKey}.stageConfig.invokes array
 *
 * @returns {} key of found stage, null if none found, or array of keys if many match
 */
module.exports.findMatchingStageShared = (serviceName, resourceType, resourceName, stages) => {

logic

1. Iterate stages to find matching stage using:
   1. match: serviceName / resourceType / resourceName
   2. For each stages.inputEventTag.properties
      1. If property's forStageMatching != false then values must match for this stage to be a match
2. if one matching stage found return its array element key
3. if find more than one matching stage return array of element keys
4. if no matching stage found return null

checkStageTestsComplete

/**
 * Tests whether any tests for a single stage are waiting results
 * @param {string} intTestTag
 * @param {number} intTestTime
 * @param {number} stageKey - array key for the stage that we just saved results for. We could skip this and check all stages, more thorough but more processing time
 *
 * @returns {boolean} true if all stage tests have results saved
 */
module.exports.checkStageTestsComplete = (stage, invokes) => {

logic

1. Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.

   (we need to query Dynamo again to prevent race conditions)

2. If no stage exists in testRecord.stages.{stageKey} add an element to testRecord.testErrors array, invoke #checkTestComplete, and return
3. Check if any tests for this stageKey remain:
   1. If the testRecord.stages.{stageKey}.stageConfig.inputEventTag set, has testRecord.stages.{stageKey}.stageResults.inputResult been set?
   2. If the testRecord.stages.{stageKey}.stageConfig.outputEventTag set, has testRecord.stages.{stageKey}.stageResults.outputResult been set?
   3. For each testRecord.stages.{stageKey}.stageConfig.invokes check has result set in testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}?
4. If all stage tests have results:
   1. Update testRecord.stages.{stageKey}.stageStatus to either passed or failed depending on whether any of the above result's resultStatus set to failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageStatus set to waiting, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stageKey}.stageErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error?)
   2. check if test complete using #checkTestComplete
5. if all stage tests have results return true, if not return false

checkTestComplete

/**
 * Tests whether any tests for a single stage are waiting results
 * @param {string} intTestTag
 * @param {number} intTestTime
 *
 * @returns {boolean} true if all stage tests have results saved
 */
module.exports.checkTestComplete = (stage, invokes) => {

logic

1. Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.

   (we need to query Dynamo again to prevent race conditions)

2. For each testRecord.stages:
   1. If any testRecord.stages.{stageKey}.stageStatus set to waiting return
   2. If no stages waiting:
      1. Update testRecord.testStatus passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.testStatus set to processing and testCompleteTimestamp to current timestamp, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.testErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error?)

testRecord.stages structure

[
    {
        stageConfig: {
            //straight copy of this stage from integration test config
        },
        stageStatus: waiting|passed|failed,
        stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stageKey}.stageStatus updated}
        stageResults: {
            //results at the point of entering the resource (eg a Lambda function is invoked)
            inputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                requestParams: {a copy of the requestParams}
            },
            //results at the point of returning from the resource (eg a Lambda function returns)
            outputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                returnValue: {a copy of the value returned by the resource} 
                //.. maybe other settings for errors etc..
            },
            //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
            invokes: {
                {serviceName_resourceType_resourceName_inputEventTag_outputEventTag}: {
                    resultTimestamp: {time result saved},
                    resultStatus: passed|failed,
                    requestParams: {a copy of the requestParams}
                },
                ..
            },
        },
        stageErrors: [
            //misc errors encountered
        ]
    },
    ...
]

Initiating tests

When the initial request is sent to begin an integration test the middleware LOG_LEVEL is set to DEBUG, this causes all Lambda functions and requests to external services to also add a message to the services MsgOut queue.

The Integration Testing service subscribes to these queues and uses these messages to monitor each stage of the workflow to ensure input and output is as expected, and that the integration test completes fully.

Add the following Correlation Ids to the initial request so we can track the integration test, filter messages, etc.:

intTest-tag

matches the test's integrationTestTag

intTest-time

matches the test's testStartedTimestamp

These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.