Service - Integration Testing

Overview

Service that pulls integration tests from the Integration Test Config service, invokes each test and monitors the steps execute as expected.

This service is intended to be run in a deployed AWS environment, all services and resources required for the integration tests need to be operational.

DynamoDB tables

TestRecords

Fields

integrationTestTag

(partition key)

testStartedTimestamp

(sort key)

time the integration test was started/created.

testCompleteTimestamp

time the integration test completed.

testSettings

straight copy of test configuration testSettings property

stages

an object that holds the details and status of each stage #testRecord.stages structure

testStatus

current status of the integration test, either processing, passed, or failed

testErrors

an array of any misc errors found (probably stored as a set, ordering is not important and should not have any duplicates)

testRecord.stages structure

[
    {
        stageConfig: {
            //straight copy of this stage from integration test config
        },
        stageStatus: waiting|passed|failed,
        stageFinishedTimestamp: {time that all tests finished and testRecord.stages.{stageKey}.stageStatus updated}
        stageResults: {
            //results at the point of entering the resource (eg a Lambda function is invoked)
            inputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                requestParams: {a copy of the requestParams}
            },
            //results at the point of returning from the resource (eg a Lambda function returns)
            outputResult: {
                resultTimestamp: {time result saved},
                resultStatus: passed|failed,
                returnValue: {a copy of the value returned by the resource} 
                //.. maybe other settings for errors etc..
            },
            //results at the point of invoking an external resource (eg the tested Lambda function is invoking another Lambda function)
            invokes: {
                {serviceName_resourceType_resourceName_inputEventTag}: {
                    invokeTimestamp: {time result saved},
                    resultTimestamp: {time result saved},
                    resultStatus: passed|failed,
                    invokeParams: {a copy of the parameters sent to the resource}
                    returnValue: {a copy of the value returned by the resource}
                },
                ..
            },
        },
        stageErrors: [
            //misc errors encountered
        ]
    },
    ...
]

Initiating tests

When the initial request is sent add the following Correlation Ids to the initial request so we can track the integration test, filter messages, etc.:

intTest-tag

matches the test's integrationTestTag

intTest-time

matches the test's testStartedTimestamp

intTest-serviceName

this services name, so other functions can generate the integration test topic names to send messages to

These can also be used in production environment to exclude requests that middleware randomly set to debug from requests initiated by Integration Test service.

The Integration Testing service monitors each stage of the workflow to ensure input and output is as expected, and that the integration test completes fully.

When a lambda receives a request that has these correlation ids it generates the integration test topic name by using the intTest-serviceName + stage + hardcoded topic name.

Ideas

More granular resource types

Currently API Gateway and SNS-SQS queue triggers for Lambda's are wrapped into the lambda event and test stage configurations, we could pull these out as separate resources that invoke the Lambda resource.

This would more clearly compartmentize each resource, increase configuration re-usablilty, and allow us to more clearly add additional tests, for example subscriptions to the SNS queues.

These new resource types could be an initial stage in the test configuration, when running integration tests the request would be sent to those resources (eg REST request to API Gateway or publish to SNS queue), on local unit tests we would need to build the triggered Lambda function directly, we could do this by following the invoke setting in the initial stage's config to find the triggered Lambda/s and building the event object for them.

Bash script to invoke initiate tests

declare -a groupTests=(
	'{ "serviceName": "Permission", "resourceType": "Lambda", "resourceName": "CheckPermission"}'
	'{ "serviceName": "Permission", "resourceType": "Lambda", "resourceName": "CheckPermission", "integrationTestTags": ["valueMatchCheck_oneRule_AnyCanPass_MustPassTrue__pass"] }'
	'{ "serviceName": "Permission", "resourceType": "Lambda", "resourceName": "CheckPermission", "integrationTestTags": ["serviceCheck__oneRule__basic__targetUserId_admin__pass"] }'
)

stage="Test"
region="us-east-2"

for test in "${groupTests[@]}"; do
	echo "=================================="
	echo "start one set of tests"
	echo ${test}
	
	#... invoke InitateTests....
	aws lambda invoke \
	--function-name "IntTesting${stage}InitiateIntegrationTestHdrInv" \
	--invocation-type Event \
	--payload "${test}" \
	--log-type Tail response.json \
	--cli-binary-format raw-in-base64-out \
	--region=${region}
done

Working documents

Working_documents - Integration Testing