Service - Integration Testing

Overview

Service that pulls integration tests from the Integration Test Config service, invokes each test and monitors the steps execute as expected.

This service is intended to be run in a deployed AWS environment, all services and resources required for the integration tests need to be operational.

DynamoDB tables

TestRecord

Fields

integrationTestTag

(partition key)

testStartedTimestamp

(sort key)

time that the integration test was started/created.

stages

an object that holds the details and status of each stage #testRecord.stages structure

testStatus

current status of the integration test, either processing, passed, or failed

errors

an array of any misc errors found

Lambda Functions

initiateIntegrationTest

/**
 * Starts integration test/s
 * @param {string} [integrationTestTag] - Only initiate test with matching integrationTestTag
 * @param {string} [serviceName] - Only initiate tests where initialStage serviceName matches
 * @param {string} [resourceType] - Only initiate tests where initialStage resourceType matches
 * @param {string} [resourceName] - Only initiate tests where initialStage resourceName matches
 *
 * @returns {boolean} true if the test was initiated successfully, false if the test could not be initiated (? or an error thrown ?)
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Invoke Service - Integration Test Config:getIntegrationTests to get configurations of all matching tests
2. For each integration test:
   1. For each integration test stage:
      1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the stages config
      2. For each invokes element:
         1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the stages invoke config

            (we could cache getEventConfig results, as many will be duplicated)

      3. Store in a variable config for the initialStage

         (we do not want to invoke the initial request until all stages are saved into TestRecord table, to avoid race conditions)

   2. Save the resulting test's config into TestRecord table, testStatus set to processing
   3. If no initialStage found add an error to testRecord.errors
   4. For the initialStage:
      1. Build initial event to start the test using the initialStage's input event
      2. Add intTest-xx correlation ids
      3. Invoke the initialStage's Lambda function

receiveMsgResourceInput

/**
 * Triggered when a resource is invoked from an integration test request
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match resource invocation, if yes perform tests and record results
 * @param {object} requestParams - The event body received by the resource
 * @param {string} serviceName
 * @param {string} resourceType
 * @param {string} resourceName
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send an system log because should not happen
2. Iterate testRecord.stages to find matching stage using:
   1. serviceName / resourceType / resourceName
   2. For each testRecord.stages.{stage}.inputEventTag.properties if forStageMatching != false values should match
3. If no matching stage found can return
4. For each testRecord.stages.{stage}.inputEventTag.properties where testValueMatches != false test if value matches
5. update testRecord.stages.{stage}.results.input values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.results.input does not already exist, if it exists add an element to testRecord.stages.{stage}.results.errors array because should only update once (message triggering receiveMsgResourceInput might get delivered multiple times, if experience that maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stage results into another table ??
6. check if any tests for this stage remain using #checkStageTestsComplete
7. if no tests remain, update testRecord.stages.{stage}.status to either passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.status set to waiting, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stage}.results.errors array (message triggering receiveMsgResourceInput might get delivered multiple times, if experience this maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stages into another table ??

receiveMsgResourceOutput

/**
 * Triggered when a resource invoked by an integration test returns
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match resource invocation, if yes perform tests on return value and record results
 * @param {object} requestParams - The event body received by the resource
 * @param {string} serviceName
 * @param {string} resourceType
 * @param {string} resourceName
 * @param {object} returnValue //how to handle errors? Maybe place the actual value into a property, and have other properties for error/other settings
*/
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

1. Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send an system log because should not happen
2. Iterate testRecord.stages to find matching stage using:
   1. serviceName / resourceType / resourceName
   2. For each testRecord.stages.{stage}.inputEventTag.properties if forStageMatching != false values should match
3. If no matching stage found can return
4. For each testRecord.stages.{stage}.inputEventTag.properties where testValueMatches != false test if value matches
5. update testRecord.stages.{stage}.results.input values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.results.input does not already exist, if it exists add an element to testRecord.stages.{stage}.results.errors array because should only update once (message triggering receiveMsgResourceInput might get delivered multiple times, if experience that maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stage results into another table ??
6. check if any tests for this stage remain using #checkStageTestsComplete
7. if no tests remain, update testRecord.stages.{stage}.status to either passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.status set to waiting, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stage}.results.errors array (message triggering receiveMsgResourceInput might get delivered multiple times, if experience this maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stages into another table ??

Functions

findMatchingStage

/**
 * Checks to see if received message has a matching stage
 * @param {object} requestParams - The event body received by the resource
 * @param {object} stages - testRecord.stages
 *
 * @returns {number} key for number array of found stage or null if none found
 */
module.exports.checkStageTestsComplete = (stage, invokes) => {

logic

1. Iterate stages to find matching stage using:
   1. serviceName / resourceType / resourceName
   2. For each testRecord.stages.{stage}.inputEventTag.properties if forStageMatching != false values should match
2. if find a matching stage return its array element key
3. if finds more than one:
   1. update testRecord.stages.{stage}.status to failed and add
   2. add error to testRecord.stages.{stage}.results.errors
4. if no matching stage found return null

checkStageTestsComplete

/**
 * Tests whether any tests for a single stage are waiting results
 * @param {object} stage - the stage object to check
 *
 * @returns {boolean} true if all stage tests have results saved
 */
module.exports.checkStageTestsComplete = (stage, invokes) => {

logic

1. check if any tests for this stage remain:
   1. if the stage.config.inputEventTag set, has stage.results.input been set?
   2. if the stage.config.outputEventTag set, has stage.results.output been set?
   3. For each stage.config.invokes check has result set in stage.results.invokes.{invoke identifier}?
2. if all stage tests have results return true

testRecord.stages structure

[
    {
        config: {
            //straight copy of this stage from integration test config
        },
        status: waiting|passed|failed,
        stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stage}.status updated}
        results: {
            input: {
                //results at the point of entering the resource (eg a Lambda function is invoked)
                timestamp: {when beginStage results were saved},
                status: passed|failed,
                requestParams: {a copy of the requestParams}
            },
            output: {
                //results at the point of entering the resource (eg a Lambda function is invoked)
                timestamp: {when beginStage results were saved},
                status: passed|failed,
                returnValue: {a copy of the value returned by the resource} 
                //.. maybe other settings for errors etc..
            },
            invokes: {
                {serviceName_resourceType_resourceName_inputEventTag_outputEventTag}: {
                    //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
                    timestamp: {when beginStage results were saved},
                    status: passed|failed,
                    requestParams: {a copy of the requestParams}
                },
                ..
            },
            errors: [
                //misc errors encountered
            ]
        }
    },
    ...
]

Initiating tests

When the initial request is sent to begin an integration test the middleware LOG_LEVEL is set to DEBUG, this causes all Lambda functions and requests to external services to also add a message to the services MsgOut queue.

The Integration Testing service subscribes to these queues and uses these messages to monitor each stage of the workflow to ensure input and output is as expected, and that the integration test completes fully.

Add the following Correlation Ids to the initial request so we can track the integration test, filter messages, etc.:

intTest-tag

matches the test's integrationTestTag

intTest-time

matches the test's testStartedTimestamp

These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.