Service - Integration Testing: Difference between revisions

From Izara Wiki
Jump to navigation Jump to search
No edit summary
Line 26: Line 26:
; testErrors
; testErrors
: an array of any misc errors found (probably stored as a set, ordering is not important and should not have any duplicates)
: an array of any misc errors found (probably stored as a set, ordering is not important and should not have any duplicates)
= Lambda Functions =
== initiateTest ==
<syntaxhighlight lang="JavaScript">
/**
* Starts integration test/s
* @param {string} [event.serviceName] - Only initiate tests where initialStage serviceName matches
* @param {string} [event.resourceType] - Only initiate tests where initialStage resourceType matches
* @param {string} [event.resourceName] - Only initiate tests where initialStage resourceName matches
* @param {string[]} [event.integrationTestTags] - Only initiate tests with matching integrationTestTags
*
* @returns {boolean} true if the test was initiated successfully, false if the test could not be initiated (? or an error thrown ?)
*/
module.exports.handler = middleware.wrap(async (event, context, callback) => {
</syntaxhighlight>
=== logic ===
# Invoke [[Service - Integration Test Config:getIntegrationTests]] to get test configurations of all matching tests
# For each integration test:
## For each integration test stage:
### Invoke [[Service - Integration Test Config:getEventConfig]] for inputEventTag, outputEventTag, record these in the test stage's config as new properties <syntaxhighlight lang="JavaScript" inline>inputEventConfig</syntaxhighlight> and <syntaxhighlight lang="JavaScript" inline>outputEventConfig</syntaxhighlight>
### For each invokes element:
#### Invoke [[Service - Integration Test Config:getEventConfig]] for inputEventTag, outputEventTag, record these in the stage invoke's config as new properties <syntaxhighlight lang="JavaScript" inline>inputEventConfig</syntaxhighlight> and <syntaxhighlight lang="JavaScript" inline>outputEventConfig</syntaxhighlight>
####: (we could cache getEventConfig results, as many will be duplicated)
### Store the config for the initialStage in a variable to be used later in logic
###: (we do not want to invoke the initial request until all stages are saved into TestRecord table, to avoid race conditions)
### if an initialStage has already been found add an error to testRecord.testErrors and set status testRecord.testStatus to failed
## If no initialStage was found add an error to testRecord.testErrors and set status testRecord.testStatus to failed
## Save the TestRecord record, including the updated testSettings, testErrors (if any), and testStatus (if not ''failed'', should be ''processing'')
## For the initialStage:
### Build initial event to start the test using the initialStage's input event
### Add intTest-xx correlation ids [[#Initiating tests]]
### Invoke the initialStage's Lambda function
== receiveMsgOutput ==
<syntaxhighlight lang="JavaScript">
/**
* Triggered when a resource invoked by an integration test returns
* Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
* Checks if any stages in integration test config match resource invocation, if yes perform tests on return value and record results
* @param {object} event.requestParams - The event body received by the resource
* @param {string} event.serviceName
* @param {string} event.resourceType
* @param {string} event.resourceName
* @param {object} event.returnValue //how to handle errors? Maybe place the actual value into a property, and have other properties for error/other settings
*/
module.exports.handler = middleware.wrap(async (event, context, callback) => {
</syntaxhighlight>
=== logic ===
# Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
# Find matching stage using [[#findMatchingStage]]
# If findMatchingStage returns false, return
# For each testRecord.stages.{stageKey}.outputEventTag.properties where testValueMatches != false test if value matches in returnValue
# ... (to update:) or straight eventValue check? ....
# ... (to update) or errors? ....
# Update testRecord.stages.{stageKey}.stageResults.outputResult values using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.outputResult does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgOutput might get delivered multiple times, if experience that maybe adjust logic to not record error?)
# Check if any tests for this stage remain using [[#checkStageTestsComplete]]
== receiveMsgInvoke ==
<syntaxhighlight lang="JavaScript">
/**
* Triggered when a resource invoked by an integration test invokes another resource
* Triggered by it's own SQS queue, which subscribes to the tested services msgOut queue
* Checks if any stages in integration test config match resource invocation, and has an invoke setting that matches message, if yes perform tests on invoke's values and record results
* @param {object} event.requestParams - The original event body received by the resource
* @param {string} event.serviceName
* @param {string} event.resourceType
* @param {string} event.resourceName
* @param {object} event.invokeRequestParams
* @param {string} event.invokeServiceName
* @param {string} event.invokeResourceType
* @param {string} event.invokeResourceName
*/
module.exports.handler = middleware.wrap(async (event, context, callback) => {
</syntaxhighlight>
=== logic ===
# Use intTest-tag and intTest-time correlation ids in received message to query TestRecord table for matching record, none found can return, but should send a system log because should not happen
# Find matching stage using [[#findMatchingStage]]
# If findMatchingStage returns false, return
# Find matching invoke for stage using [[#findMatchingStageShared]], inputting testRecord.stages.{stageKey}.stageConfig.invokes and '''invoke..''' params
# If no matching invoke found and testRecord.testSettings.errorIfInvokeUndefined set to true:
## Add error to testRecord.stages.{stageKey}.stageErrors
## Save testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}, resultStatus set to '''failed'''
## Check if any stageResults for this stage remain using [[#checkStageTestsComplete]]
## Return false
# If more than one matching invoke found:
## Add error to testRecord.stages.{stageKey}.stageErrors
## For each invoke found:
### Save testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}, resultStatus set to '''failed'''
## Check if any stageResults for this stage remain using [[#checkStageTestsComplete]]
## Return false
# If message is type '''invokeResourceRequest'''
## update testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}.invokeTimestamp and testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}.invokeParams using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag} does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgInvoke might get delivered multiple times, if experience that maybe adjust logic to not record error?)
# If message is type '''invokeResourceResponse'''
## For each testRecord.stages.{stageKey}.stageConfig.invokes.{invokeKey}.outputEventTag.properties where testValueMatches != false test if value matches in invokeRequestParams
# ... (to update) or straight value check ....
# ... (to update) or errors? ....
# Update testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}.resultTimestamp / resultStatus / returnValue using a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag} does not already exist, if it exists add an element to testRecord.stages.{stageKey}.stageErrors array because should only update once (message triggering receiveMsgInvoke might get delivered multiple times, if experience that maybe adjust logic to not record error?)
# Check if any tests for this stage remain using [[#checkStageTestsComplete]]
= Functions =
== findMatchingStage ==
<syntaxhighlight lang="JavaScript">
/**
* Checks to see if received message has a matching stage
* @param {string} serviceName
* @param {string} resourceType
* @param {string} resourceName
* @param {object} inputEventParams - parameters that the resource received as its request, these are compared against stage config to find matching stage
* @param {object} testRecord - testRecord's configuration
* @param {string} resultTag - matches msgTag, whether is input or output message, to make errors more detailed
*
* @returns {number} key of found stage or null if none found
*/
module.exports.findMatchingStage = (serviceName, resourceType, resourceName, inputEventParams, testRecord, resultTag) => {
</syntaxhighlight>
=== logic ===
# find matching stage/s using [[#findMatchingStageShared]], pass in testRecord.stages
# If no matching stage found and testRecord.errorIfStageUndefined set to true:
## update testRecord database record, adding an element to testRecord.testErrors (can use UpdateItem.Add method so not need to worry about race condition)
## Return false
# If more than one matching stage found, for each found stage:
## update testRecord database record, adding an element to testRecord.stages.{stageKey}.stageErrors and add/update testRecord.stages.{stageKey}.stageResults.[resultTag], resultStatus set to '''failed''' (probably skip saving additional fields like requestParams and returnValue, is an edge case)(don't worry about race conditions, should not cause serious issues)
## Check if any stageResults for this stage remain using [[#checkStageTestsComplete]]
## Return false
# no errors can return element key of found stage
== findMatchingStageShared ==
<syntaxhighlight lang="JavaScript">
/**
* Checks to see if received message has a matching stage
* @param {string} serviceName
* @param {string} resourceType
* @param {string} resourceName
* @param {object} inputEventParams - parameters that the resource received as its request, these are compared against stage config to find matching stage
* @param {object} stages - either testRecord.stages array or testRecord.stages.{stageKey}.stageConfig.invokes array
*
* @returns {} key of found stage, null if none found, or array of keys if many match
*/
module.exports.findMatchingStageShared = (serviceName, resourceType, resourceName, inputEventParams, stages) => {
</syntaxhighlight>
=== logic ===
# Iterate stages to find matching stage using:
## match: serviceName / resourceType / resourceName
## For each stages.inputEventTag.properties
### If property's forStageMatching != false then values must match for this stage to be a match
# if one matching stage found return its array element key
# if find more than one matching stage return array of element keys
# if no matching stage found return null
== checkStageTestsComplete ==
<syntaxhighlight lang="JavaScript">
/**
* Tests whether any tests for a single stage are waiting results
* @param {string} intTestTag
* @param {number} intTestTime
* @param {number} stageKey - array key for the stage that we just saved results for. We could skip this and check all stages, more thorough but more processing time
*
* @returns {boolean} true if all stage tests have results saved
*/
module.exports.checkStageTestsComplete = (intTestTag, intTestTime, stageKey) => {
</syntaxhighlight>
=== logic ===
# Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.
#: (we need to query Dynamo again even though calling code probably already has TestRecord, to prevent race conditions)
# If no stage exists in testRecord.stages.{stageKey}:
## update testRecord database record, adding an element to testRecord.testErrors (can use UpdateItem.Add method)
## invoke [[#checkTestComplete]]
## return true(?)
# Check if any tests for this stageKey remain:
## If the testRecord.stages.{stageKey}.stageConfig.inputEventTag set, has testRecord.stages.{stageKey}.stageResults.inputResult been set?
## If the testRecord.stages.{stageKey}.stageConfig.outputEventTag set, has testRecord.stages.{stageKey}.stageResults.outputResult been set?
## For each testRecord.stages.{stageKey}.stageConfig.invokes check has result set in testRecord.stages.{stageKey}.stageResults.invokes.{invokeTag}?
## if any results not yet been can return false
# If all stage tests have results:
## Update testRecord.stages.{stageKey}.stageStatus to either passed or failed depending on whether any of the above result's resultStatus set to '''failed''', use a DynamoDB Conditional Expression to make sure testRecord.stages.{stageKey}.stageStatus set to '''waiting''', if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.stages.{stageKey}.stageErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error)
## check if test complete using [[#checkTestComplete]]
# all stage tests have results so return true
== checkTestComplete ==
<syntaxhighlight lang="JavaScript">
/**
* Tests whether any tests for a single stage are waiting results
* @param {string} intTestTag
* @param {number} intTestTime
*
* @returns {boolean} true if all stage tests have results saved
*/
module.exports.checkTestComplete = (intTestTag, intTestTime) => {
</syntaxhighlight>
=== logic ===
# Use intTestTag and intTestTime to query TestRecord table for matching record, none found can return, but should send a system log because should not happen.
#: (we need to query Dynamo again even though calling code probably has TestRecord already, to prevent race conditions)
# For each testRecord.stages:
## If any testRecord.stages.{stageKey}.stageStatus set to '''waiting''' return
# If no stages '''waiting''':
## Update testRecord.testStatus '''passed''' or '''failed''', use a DynamoDB Conditional Expression to make sure testRecord.testStatus set to '''processing''' and testCompleteTimestamp to current timestamp, if conditional expression does not pass means another process already updated it, should not happen, add an element to testRecord.testErrors array (message triggering this processing might get delivered multiple times, if experience this maybe adjust logic to not record error)


= testRecord.stages structure =
= testRecord.stages structure =
Line 305: Line 86:
These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.
These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.


= Working documents =
[[:Category:Working_documents|Working documents]]


[[Category:Backend services| Integration Testing]]
[[Category:Backend services| Integration Testing]]

Revision as of 08:41, 4 November 2020

Overview

Service that pulls integration tests from the Integration Test Config service, invokes each test and monitors the steps execute as expected.

This service is intended to be run in a deployed AWS environment, all services and resources required for the integration tests need to be operational.

DynamoDB tables

TestRecord

Fields

integrationTestTag
(partition key)
testStartedTimestamp
(sort key)
time the integration test was started/created.
testCompleteTimestamp
time the integration test completed.
testSettings
straight copy of test configuration testSettings property
stages
an object that holds the details and status of each stage #testRecord.stages structure
testStatus
current status of the integration test, either processing, passed, or failed
testErrors
an array of any misc errors found (probably stored as a set, ordering is not important and should not have any duplicates)

testRecord.stages structure

[
    {
        stageConfig: {
            //straight copy of this stage from integration test config
        },
        stageStatus: waiting|passed|failed,
        stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stageKey}.stageStatus updated}
        stageResults: {
            //results at the point of entering the resource (eg a Lambda function is invoked)
            inputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                requestParams: {a copy of the requestParams}
            },
            //results at the point of returning from the resource (eg a Lambda function returns)
            outputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                returnValue: {a copy of the value returned by the resource} 
                //.. maybe other settings for errors etc..
            },
            //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
            invokes: {
                {serviceName_resourceType_resourceName_inputEventTag}: {
                    invokeTimestamp: {time result saved},
                    resultTimestamp: {time result saved},
                    resultStatus: passed|failed,
                    invokeParams: {a copy of the parameters sent to the resource}
                    returnValue: {a copy of the value returned by the resource}
                },
                ..
            },
        },
        stageErrors: [
            //misc errors encountered
        ]
    },
    ...
]

Initiating tests

When the initial request is sent to begin an integration test the middleware LOG_LEVEL is set to DEBUG, this causes all Lambda functions and requests to external services to also add a message to the services MsgOut queue.

The Integration Testing service subscribes to these queues and uses these messages to monitor each stage of the workflow to ensure input and output is as expected, and that the integration test completes fully.

Add the following Correlation Ids to the initial request so we can track the integration test, filter messages, etc.:

intTest-tag
matches the test's integrationTestTag
intTest-time
matches the test's testStartedTimestamp

These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.

Working documents

Working documents

Retrieved from "https://izara.io/wiki/index.php?title=Service_-_Integration_Testing&oldid=276"