Service - Integration Testing

Overview

Service that pulls integration tests from the Integration Test Config service, invokes each test and monitors the steps execute as expected.

This service is intended to be run in a deployed AWS environment, all services and resources required for the integration tests need to be operational.

DynamoDB tables

TestRecord

Fields

integrationTestTag

(partition key)

testStartedTimestamp

(sort key)

time that the integration test was started/created.

stages

an object that holds the details and status of each stage #testRecord.stages structure

testStatus

current status of the integration test, either processing, passed, or failed

errors

an array of any misc errors found

Lambda Functions

initiateIntegrationTest

/**
 * Starts integration test/s
 * @param {string} [integrationTestTag] - Only initiate test with matching integrationTestTag
 * @param {string} [serviceName] - Only initiate tests where initialStage serviceName matches
 * @param {string} [resourceType] - Only initiate tests where initialStage resourceType matches
 * @param {string} [resourceName] - Only initiate tests where initialStage resourceName matches
 *
 * @returns {boolean} true if the test was initiated successfully, false if the test could not be initiated (? or an error thrown ?)
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Invoke Service - Integration Test Config:getIntegrationTests to get configurations of all matching tests
2. For each integration test:
   1. For each integration test stage:
      1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the stages config
      2. For each invokes element:
         1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the stages invoke config

            (we could cache getEventConfig results, as many will be duplicated)

      3. Store in a variable config for the initialStage

         (we do not want to invoke the initial request until all stages are saved into TestRecord table, to avoid race conditions)

   2. Save the resulting test's config into TestRecord table, testStatus set to processing
   3. If no initialStage found add an error to testRecord.errors
   4. For the initialStage:
      1. Build initial event to start the test using the initialStage's input event
      2. Add intTest-xx correlation ids
      3. Invoke the initialStage's Lambda function

receiveMsgLambdaBegin

/**
 * Triggered when a Lambda function is invoked from an integration test request
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match Lambda invocation, if yes perform tests and record results
 * @param {object} requestProperties - The event body for the invoked Lambda
 * @param {string} serviceName - Name of the service that owns the Lambda function, matches stage's serviceName
 * @param {string} functionName - Name of the invoked Lambda function, matches stage's resourceName
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send an system log because should not happen
2. Iterate testRecord.stages to find matching stage using:
   1. serviceName / resourceType (Lambda) / resourceName
   2. For each testRecord.stages.{stage}.inputEventTag.properties if forStageMatching != false values should match
3. If no matching stage found can return
4. For each testRecord.stages.{stage}.inputEventTag.properties where testValueMatches != false test if value matches
5. update testRecord.stages.{stage}.results.input values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.results.input does not already exist, if it exists add an element to testRecord.stages.{stage}.results.errors array because should only update once (message triggering receiveMsgLambdaBegin might get delivered multiple times, if experience that maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stage results into another table ??
6. check if any tests for this stage remain using #checkStageTestsComplete
7. if no tests remain, update testRecord.stages.{stage}.status to either passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.status set to waiting, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stage}.results.errors array (message triggering receiveMsgLambdaBegin might get delivered multiple times, if experience this maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stages into another table ??

receiveMsgLambdaEnd

/**
 * Triggered when a Lambda function invoked by an integration test request returns
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match Lambda invocation, if yes perform tests and record results
 * @param {object} requestProperties - The event body for the invoked Lambda
 * @param {string} serviceName - Name of the service that owns the Lambda function, matches stage's serviceName
 * @param {string} functionName - Name of the invoked Lambda function, matches stage's resourceName
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for test record
2. iterate stages to find match
3. if no match found can return
4. Invoke Service - Integration Test Config:getEventConfig to pull inputEventTag config
5. for each properties in event config test if value matches
6. update testRecord.stages.{stage}.results.input values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.results.input does not already exist, if it exists add an element to testRecord.stages.{stage}.results.errors array because should only update once (message triggering receiveMsgLambdaBegin might get delivered multiple times, if experience that maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stage results into another table ??
7. check if any tests for this stage remain:
   1. if the stage has outputEventTag set, has testRecord.stages.{stage}.results.output been set?
   2. if the stage has invokes set, does each invoke element have results set in testRecord.stages.{stage}.results.invokes.{invoke identifier} been set?
8. if no tests remain, update testRecord.stages.{stage}.status to either passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.status set to waiting, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stage}.results.errors array (message triggering receiveMsgLambdaBegin might get delivered multiple times, if experience this maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stages into another table ??

Functions

checkStageTestsComplete

/**
 * Tests whether any tests for a single stage are waiting results
 * @param {object} stage - the stage object to check
 *
 * @returns {boolean} true if all stage tests have results saved
 */
module.exports.checkStageTestsComplete = (stage, invokes) => {

logic

1. check if any tests for this stage remain:
   1. if the stage.config.inputEventTag set, has stage.results.input been set?
   2. if the stage.config.outputEventTag set, has stage.results.output been set?
   3. For each stage.config.invokes check has result set in stage.results.invokes.{invoke identifier}?
2. if all stage tests have results return true

testRecord.stages structure

[
    {
        config: {
            //straight copy of this stage from integration test config
        },
        status: waiting|passed|failed,
        stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stage}.status updated}
        results: {
            input: {
                //results at the point of entering the resource (eg a Lambda function is invoked)
                timestamp: {when beginStage results were saved},
                status: passed|failed,
                requestProperties: {a copy of the requestProperties}
            },
            output: {
                //results at the point of entering the resource (eg a Lambda function is invoked)
                timestamp: {when beginStage results were saved},
                status: passed|failed,
                returnValue: {a copy of the value returned by the resource} 
                //.. maybe other settings for errors etc..
            },
            invokes: {
                {serviceName_resourceType_resourceName_inputEventTag_outputEventTag}: {
                    //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
                    timestamp: {when beginStage results were saved},
                    status: passed|failed,
                    requestProperties: {a copy of the requestProperties}
                },
                ..
            },
            errors: [
                //misc errors encountered
            ]
        }
    },
    ...
]

Initiating tests

When the initial request is sent to begin an integration test the middleware LOG_LEVEL is set to DEBUG, this causes all Lambda functions and requests to external services to also add a message to the services MsgOut queue.

The Integration Testing service subscribes to these queues and uses these messages to monitor each stage of the workflow to ensure input and output is as expected, and that the integration test completes fully.

Add the following Correlation Ids to the initial request so we can track the integration test, filter messages, etc.:

intTest-tag

matches the test's integrationTestTag

intTest-time

matches the test's testStartedTimestamp

These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.