Service - Integration Testing: Difference between revisions

From Izara Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 15: Line 15:
; testStartedTimestamp
; testStartedTimestamp
: (sort key)
: (sort key)
: time that the integration test was started/created.
: time the integration test was started/created.
; testCompleteTimestamp
: time the integration test completed.
; stages
; stages
: an object that holds the details and status of each stage [[#testRecord.stages structure]]
: an object that holds the details and status of each stage [[#testRecord.stages structure]]
; testStatus
; testStatus
: current status of the integration test, either '''processing''', '''passed''', or '''failed'''
: current status of the integration test, either '''processing''', '''passed''', or '''failed'''
; errors
; testErrors
: an array of any misc errors found
: an array of any misc errors found


Line 52: Line 54:
###: (we do not want to invoke the initial request until all stages are saved into TestRecord table, to avoid race conditions)
###: (we do not want to invoke the initial request until all stages are saved into TestRecord table, to avoid race conditions)
## Save the resulting test's config into TestRecord table, testStatus set to ''processing''
## Save the resulting test's config into TestRecord table, testStatus set to ''processing''
## If no initialStage found add an error to testRecord.errors
## If no initialStage found add an error to testRecord.testErrors
## For the initialStage:
## For the initialStage:
### Build initial event to start the test using the initialStage's input event
### Build initial event to start the test using the initialStage's input event
Line 75: Line 77:
=== logic ===
=== logic ===


# Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send an system log because should not happen
# Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
# Iterate testRecord.stages to find matching stage using:
# Find matching stage using [[#findMatchingStage]]
## serviceName / resourceType / resourceName
# If no matching stage found:
## For each testRecord.stages.{stage}.inputEventTag.properties if forStageMatching != false values should match
## If errorIfStageUndefined in test configuration set to true, record an error
# If no matching stage found can return
## Return
# For each testRecord.stages.{stage}.inputEventTag.properties where testValueMatches != false test if value matches
# For each testRecord.stages.{stageKey}.inputEventTag.properties where testValueMatches != false test if value matches in requestParams
# update testRecord.stages.{stage}.results.input values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.results.input does not already exist, if it exists add an element to testRecord.stages.{stage}.results.errors array because should only update once (message triggering receiveMsgResourceInput might get delivered multiple times, if experience that maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stage results into another table ??
# ... (to update) or straight value check ....
# check if any tests for this stage remain using [[#checkStageTestsComplete]]
# Update testRecord.stages.{stageKey}.stageResults.inputResult values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.inputResult does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgResourceInput might get delivered multiple times, if experience that maybe adjust logic to not record error?)
# if no tests remain, update testRecord.stages.{stage}.status to either passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.status set to '''waiting''', if conditional expression does not pass means another process already updated it, should not happen,  add an element to testRecord.stages.{stage}.results.errors array (message triggering receiveMsgResourceInput might get delivered multiple times, if experience this maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stages into another table ??
# Check if any tests for this stage remain using [[#checkStageTestsComplete]]


== receiveMsgResourceOutput ==
== receiveMsgResourceOutput ==
Line 103: Line 105:
=== logic ===
=== logic ===


# Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send an system log because should not happen
# Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
# Iterate testRecord.stages to find matching stage using:
# Find matching stage using [[#findMatchingStage]]
## serviceName / resourceType / resourceName
# If no matching stage found:
## For each testRecord.stages.{stage}.inputEventTag.properties if forStageMatching != false values should match
## If errorIfStageUndefined in test configuration set to true, record an error
# If no matching stage found can return
## Return
# For each testRecord.stages.{stage}.inputEventTag.properties where testValueMatches != false test if value matches
# For each testRecord.stages.{stageKey}.inputEventTag.properties where testValueMatches != false test if value matches in returnValue
# update testRecord.stages.{stage}.results.input values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.results.input does not already exist, if it exists add an element to testRecord.stages.{stage}.results.errors array because should only update once (message triggering receiveMsgResourceInput might get delivered multiple times, if experience that maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stage results into another table ??
# ... (to update) or straight value check ....
# check if any tests for this stage remain using [[#checkStageTestsComplete]]
# ... (to update) or errors? ....
# if no tests remain, update testRecord.stages.{stage}.status to either passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stage}.status set to '''waiting''', if conditional expression does not pass means another process already updated it, should not happen,  add an element to testRecord.stages.{stage}.results.errors array (message triggering receiveMsgResourceInput might get delivered multiple times, if experience this maybe adjust logic) ?? not sure can conditional expression on nested property in JSON, probably not, might need to move stages into another table ??
# Update testRecord.stages.{stageKey}.stageResults.outputResult values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.outputResult does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgResourceOutput might get delivered multiple times, if experience that maybe adjust logic to not record error?)
# Check if any tests for this stage remain using [[#checkStageTestsComplete]]


= Functions =
= Functions =
Line 123: Line 126:
  * @param {object} stages - testRecord.stages
  * @param {object} stages - testRecord.stages
  *
  *
  * @returns {number} key for number array of found stage or null if none found
  * @returns {number} key of found stage or null if none found
  */
  */
module.exports.checkStageTestsComplete = (stage, invokes) => {
module.exports.findMatchingStage = (stage, invokes) => {
</syntaxhighlight>
</syntaxhighlight>


Line 132: Line 135:
# Iterate stages to find matching stage using:
# Iterate stages to find matching stage using:
## serviceName / resourceType / resourceName
## serviceName / resourceType / resourceName
## For each testRecord.stages.{stage}.inputEventTag.properties if forStageMatching != false values should match
## For each testRecord.stages.{stageKey}.inputEventTag.properties if forStageMatching != false values must match
# if find a matching stage return its array element key
# if matching stage found return its array element key
# if finds more than one:
# if find more than one matching stage:
## update testRecord.stages.{stage}.status to failed and add
## update testRecord.stages.{stageKey}.stageStatus to failed
## add error to testRecord.stages.{stage}.results.errors
## add error to testRecord.stages.{stageKey}.stageErrors
# if no matching stage found return null
# if no matching stage found return null


Line 144: Line 147:
/**
/**
  * Tests whether any tests for a single stage are waiting results
  * Tests whether any tests for a single stage are waiting results
  * @param {object} stage - the stage object to check
  * @param {string} intTestTag
* @param {number} intTestTime
* @param {number} stageKey - array key for the stage that we just saved results for. We could skip this and check all stages, more thorough but more processing time
  *
  *
  * @returns {boolean} true if all stage tests have results saved
  * @returns {boolean} true if all stage tests have results saved
Line 153: Line 158:
=== logic ===
=== logic ===


# check if any tests for this stage remain:
# Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.
## if the stage.config.inputEventTag set, has stage.results.input been set?
#: (we need to query Dynamo again to prevent race conditions)
## if the stage.config.outputEventTag set, has stage.results.output been set?
# If no stage exists in testRecord.stages.{stageKey} add an element to testRecord.testErrors array, invoke [[#checkTestComplete]], and return
## For each stage.config.invokes check has result set in stage.results.invokes.{invoke identifier}?
# Check if any tests for this stageKey remain:
# if all stage tests have results return true
## If the testRecord.stages.{stageKey}.stageConfig.inputEventTag set, has testRecord.stages.{stageKey}.stageResults.inputResult been set?
## If the testRecord.stages.{stageKey}.stageConfig.outputEventTag set, has testRecord.stages.{stageKey}.stageResults.outputResult been set?
## For each testRecord.stages.{stageKey}.stageConfig.invokes check has result set in testRecord.stages.{stageKey}.stageResults.invokes.{invoke identifier}?
# If all stage tests have results:
## Update testRecord.stages.{stageKey}.stageStatus to either passed or failed depending on whether any of the above result's resultStatus set to '''failed''', use a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageStatus set to '''waiting''', if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stageKey}.stageErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error?)
## check if test complete using [[#checkTestComplete]]
# if all stage tests have results return true, if not return false
 
== checkTestComplete ==
 
<syntaxhighlight lang="JavaScript">
/**
* Tests whether any tests for a single stage are waiting results
* @param {string} intTestTag
* @param {number} intTestTime
*
* @returns {boolean} true if all stage tests have results saved
*/
module.exports.checkTestComplete = (stage, invokes) => {
</syntaxhighlight>
 
=== logic ===
 
# Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.
#: (we need to query Dynamo again to prevent race conditions)
# For each testRecord.stages:
## If any testRecord.stages.{stageKey}.stageStatus set to '''waiting''' return
## If no stages '''waiting''':
### Update testRecord.testStatus '''passed''' or '''failed''', use a DynamoDB Conditional Expression to make sure testRecord.testStatus set to '''processing''' and testCompleteTimestamp to current timestamp, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.testErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error?)


= testRecord.stages structure =
= testRecord.stages structure =
Line 164: Line 197:
[
[
     {
     {
         config: {
         stageConfig: {
             //straight copy of this stage from integration test config
             //straight copy of this stage from integration test config
         },
         },
         status: waiting|passed|failed,
         stageStatus: waiting|passed|failed,
         stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stage}.status updated}
         stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stageKey}.stageStatus updated}
         results: {
         stageResults: {
             input: {
             //results at the point of entering the resource (eg a Lambda function is invoked)
                //results at the point of entering the resource (eg a Lambda function is invoked)
            inputResult: {
                 timestamp: {when beginStage results were saved},
                 resultTimestamp: {time result saved},
                 status: passed|failed,
                 resultStatus: passed|failed,
                 requestParams: {a copy of the requestParams}
                 requestParams: {a copy of the requestParams}
             },
             },
             output: {
             //results at the point of returning from the resource (eg a Lambda function returns)
                //results at the point of entering the resource (eg a Lambda function is invoked)
            outputResult: {
                 timestamp: {when beginStage results were saved},
                 resultTimestamp: {time result saved},
                 status: passed|failed,
                 resultStatus: passed|failed,
                 returnValue: {a copy of the value returned by the resource}  
                 returnValue: {a copy of the value returned by the resource}  
                 //.. maybe other settings for errors etc..
                 //.. maybe other settings for errors etc..
             },
             },
            //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
             invokes: {
             invokes: {
                 {serviceName_resourceType_resourceName_inputEventTag_outputEventTag}: {
                 {serviceName_resourceType_resourceName_inputEventTag_outputEventTag}: {
                     //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
                     resultTimestamp: {time result saved},
                    timestamp: {when beginStage results were saved},
                     resultStatus: passed|failed,
                     status: passed|failed,
                     requestParams: {a copy of the requestParams}
                     requestParams: {a copy of the requestParams}
                 },
                 },
                 ..
                 ..
             },
             },
            errors: [
        },
                //misc errors encountered
        stageErrors: [
            ]
            //misc errors encountered
        }
        ]
     },
     },
     ...
     ...

Revision as of 12:09, 6 September 2020

Overview

Service that pulls integration tests from the Integration Test Config service, invokes each test and monitors the steps execute as expected.

This service is intended to be run in a deployed AWS environment, all services and resources required for the integration tests need to be operational.

DynamoDB tables

TestRecord

Fields

integrationTestTag
(partition key)
testStartedTimestamp
(sort key)
time the integration test was started/created.
testCompleteTimestamp
time the integration test completed.
stages
an object that holds the details and status of each stage #testRecord.stages structure
testStatus
current status of the integration test, either processing, passed, or failed
testErrors
an array of any misc errors found

Lambda Functions

initiateIntegrationTest

/**
 * Starts integration test/s
 * @param {string} [integrationTestTag] - Only initiate test with matching integrationTestTag
 * @param {string} [serviceName] - Only initiate tests where initialStage serviceName matches
 * @param {string} [resourceType] - Only initiate tests where initialStage resourceType matches
 * @param {string} [resourceName] - Only initiate tests where initialStage resourceName matches
 *
 * @returns {boolean} true if the test was initiated successfully, false if the test could not be initiated (? or an error thrown ?)
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

  1. Invoke Service - Integration Test Config:getIntegrationTests to get configurations of all matching tests
  2. For each integration test:
    1. For each integration test stage:
      1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the stages config
      2. For each invokes element:
        1. Invoke Service - Integration Test Config:getEventConfig for inputEventTag, outputEventTag, record these in the stages invoke config
          (we could cache getEventConfig results, as many will be duplicated)
      3. Store in a variable config for the initialStage
        (we do not want to invoke the initial request until all stages are saved into TestRecord table, to avoid race conditions)
    2. Save the resulting test's config into TestRecord table, testStatus set to processing
    3. If no initialStage found add an error to testRecord.testErrors
    4. For the initialStage:
      1. Build initial event to start the test using the initialStage's input event
      2. Add intTest-xx correlation ids
      3. Invoke the initialStage's Lambda function

receiveMsgResourceInput

/**
 * Triggered when a resource is invoked from an integration test request
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match resource invocation, if yes perform tests and record results
 * @param {object} requestParams - The event body received by the resource
 * @param {string} serviceName
 * @param {string} resourceType
 * @param {string} resourceName
 */
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

  1. Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
  2. Find matching stage using #findMatchingStage
  3. If no matching stage found:
    1. If errorIfStageUndefined in test configuration set to true, record an error
    2. Return
  4. For each testRecord.stages.{stageKey}.inputEventTag.properties where testValueMatches != false test if value matches in requestParams
  5. ... (to update) or straight value check ....
  6. Update testRecord.stages.{stageKey}.stageResults.inputResult values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.inputResult does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgResourceInput might get delivered multiple times, if experience that maybe adjust logic to not record error?)
  7. Check if any tests for this stage remain using #checkStageTestsComplete

receiveMsgResourceOutput

/**
 * Triggered when a resource invoked by an integration test returns
 * Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
 * Checks if any stages in integration test config match resource invocation, if yes perform tests on return value and record results
 * @param {object} requestParams - The event body received by the resource
 * @param {string} serviceName
 * @param {string} resourceType
 * @param {string} resourceName
 * @param {object} returnValue //how to handle errors? Maybe place the actual value into a property, and have other properties for error/other settings
*/
module.exports.handler = middleware.wrap(async (event, context, callback) => {

logic

  1. Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
  2. Find matching stage using #findMatchingStage
  3. If no matching stage found:
    1. If errorIfStageUndefined in test configuration set to true, record an error
    2. Return
  4. For each testRecord.stages.{stageKey}.inputEventTag.properties where testValueMatches != false test if value matches in returnValue
  5. ... (to update) or straight value check ....
  6. ... (to update) or errors? ....
  7. Update testRecord.stages.{stageKey}.stageResults.outputResult values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.outputResult does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgResourceOutput might get delivered multiple times, if experience that maybe adjust logic to not record error?)
  8. Check if any tests for this stage remain using #checkStageTestsComplete

Functions

findMatchingStage

/**
 * Checks to see if received message has a matching stage
 * @param {object} requestParams - The event body received by the resource
 * @param {object} stages - testRecord.stages
 *
 * @returns {number} key of found stage or null if none found
 */
module.exports.findMatchingStage = (stage, invokes) => {

logic

  1. Iterate stages to find matching stage using:
    1. serviceName / resourceType / resourceName
    2. For each testRecord.stages.{stageKey}.inputEventTag.properties if forStageMatching != false values must match
  2. if matching stage found return its array element key
  3. if find more than one matching stage:
    1. update testRecord.stages.{stageKey}.stageStatus to failed
    2. add error to testRecord.stages.{stageKey}.stageErrors
  4. if no matching stage found return null

checkStageTestsComplete

/**
 * Tests whether any tests for a single stage are waiting results
 * @param {string} intTestTag
 * @param {number} intTestTime
 * @param {number} stageKey - array key for the stage that we just saved results for. We could skip this and check all stages, more thorough but more processing time
 *
 * @returns {boolean} true if all stage tests have results saved
 */
module.exports.checkStageTestsComplete = (stage, invokes) => {

logic

  1. Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.
    (we need to query Dynamo again to prevent race conditions)
  2. If no stage exists in testRecord.stages.{stageKey} add an element to testRecord.testErrors array, invoke #checkTestComplete, and return
  3. Check if any tests for this stageKey remain:
    1. If the testRecord.stages.{stageKey}.stageConfig.inputEventTag set, has testRecord.stages.{stageKey}.stageResults.inputResult been set?
    2. If the testRecord.stages.{stageKey}.stageConfig.outputEventTag set, has testRecord.stages.{stageKey}.stageResults.outputResult been set?
    3. For each testRecord.stages.{stageKey}.stageConfig.invokes check has result set in testRecord.stages.{stageKey}.stageResults.invokes.{invoke identifier}?
  4. If all stage tests have results:
    1. Update testRecord.stages.{stageKey}.stageStatus to either passed or failed depending on whether any of the above result's resultStatus set to failed, use a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageStatus set to waiting, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stageKey}.stageErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error?)
    2. check if test complete using #checkTestComplete
  5. if all stage tests have results return true, if not return false

checkTestComplete

/**
 * Tests whether any tests for a single stage are waiting results
 * @param {string} intTestTag
 * @param {number} intTestTime
 *
 * @returns {boolean} true if all stage tests have results saved
 */
module.exports.checkTestComplete = (stage, invokes) => {

logic

  1. Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.
    (we need to query Dynamo again to prevent race conditions)
  2. For each testRecord.stages:
    1. If any testRecord.stages.{stageKey}.stageStatus set to waiting return
    2. If no stages waiting:
      1. Update testRecord.testStatus passed or failed, use a DynamoDB Conditional Expression to make sure testRecord.testStatus set to processing and testCompleteTimestamp to current timestamp, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.testErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error?)

testRecord.stages structure

[
    {
        stageConfig: {
            //straight copy of this stage from integration test config
        },
        stageStatus: waiting|passed|failed,
        stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stageKey}.stageStatus updated}
        stageResults: {
            //results at the point of entering the resource (eg a Lambda function is invoked)
            inputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                requestParams: {a copy of the requestParams}
            },
            //results at the point of returning from the resource (eg a Lambda function returns)
            outputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                returnValue: {a copy of the value returned by the resource} 
                //.. maybe other settings for errors etc..
            },
            //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
            invokes: {
                {serviceName_resourceType_resourceName_inputEventTag_outputEventTag}: {
                    resultTimestamp: {time result saved},
                    resultStatus: passed|failed,
                    requestParams: {a copy of the requestParams}
                },
                ..
            },
        },
        stageErrors: [
            //misc errors encountered
        ]
    },
    ...
]

Initiating tests

When the initial request is sent to begin an integration test the middleware LOG_LEVEL is set to DEBUG, this causes all Lambda functions and requests to external services to also add a message to the services MsgOut queue.

The Integration Testing service subscribes to these queues and uses these messages to monitor each stage of the workflow to ensure input and output is as expected, and that the integration test completes fully.

Add the following Correlation Ids to the initial request so we can track the integration test, filter messages, etc.:

intTest-tag
matches the test's integrationTestTag
intTest-time
matches the test's testStartedTimestamp

These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.