2019-01-01 - Mocking AWS resources in local test environment

From Izara Wiki
Jump to navigation Jump to search

Local Environment

Originally was looking at setting up environment that would allow requests to pass between locally deployed services to test workflows (mainly SNS>SQS>Lambda triggers)

Decided to pull back to unit tests only because:

  1. modules to emulate AWS services locally are not all well maintained (see notes eg SNS offline) or lack functionality
  2. the complexity in running a large number of services locally to test code is high, also some comments that it could hog resources development machine

If we took above path it could break easily when any part is not maintained, and creates a high workload for developers to fire up/maintain test environment

Current thoughts

  • Only doing direct invoke (not via lambda, but by including the code) tests, full AWS service flow is not possible local, and setting up testing features like mock probably messy/not possible in serverless offline environment
  • Can do unit tests (one function) and basic integration tests that do not involve lambda/queues etc.

Start Request: Lambda triggered by API

  • Jest direct invoke node lambda handler
  • maybe test initial logic function this way too, rather than having 2 unit tests, one for the handler and one for the logic

Start Request: Lambda triggered by SQS

  • Jest direct invoke node lambda handler
  • request formatted as batch Records
  • maybe test initial logic function this way too, rather than having 2 unit tests, one for the handler and one for the logic

Start Request: Lambda triggered by direct invoke

  • Jest direct invoke lambda
  • if lambda is designed to be a sync invoke, then it will likely return a value
  • if designed to be async invoke then other outputs would be expected, eg messages sent to (mocked) queue, or Dynamo data updated.
  • For Dynamo data updated from an async invoked lambda, when testing maybe we can invoke sync which will force Jest to wait for execution to complete before checking Dynamo results

Start Request: Other (eg Shared) functions

  • Jest direct invoke node

Check Response: Lambda API Return Value

  • Jest receives response from the REST request
  • If lambda being tested is designed to invoke async means the calling code does not wait for response. We can perhaps force Jest to invokes sync so Jest waits for execution to complete before checking results (eg in a Dynamo table)

Check Response: adjust Dynamo data inside requested function

  • Jest checks database after API request returns
  • I understand the lambda function should not return until all async operations are complete, otherwise lambda cleanup can dump anything remaining in call stack, so the data should be changed before Jest gets the response from REST request
  • (an alternative we probably do not need to use is a Dynamo stream triggering Jest)

Check Response: send message to SNS queue

  • Jest mocks SNS queue
  • send message request from lambda returns success
  • message gets received into Jest for testing

Check Response: direct invoke another Lambda within existing service

  • We could either mock the lambda.invoke call so the target lambda never gets invoked, and pass back expected response (if invoking sync). Also Jest can test the lambda.invoke request is as expected
  • Maybe could mock the invoke so the resulting lambda’s code gets called directly

Check Response: Lambda direct invokes another Lambda outside existing service

  • probably mock the lambda.invoke call so the target lambda never gets invoked
  • if it is a sync invocation will likely expect a result, will need to build that into mock/test config

Local Modules

DynamoDB-Local

copied to Local test environment#DynamoDB Local

DynamoDB local connection settings are hardcoded, if need to change things like port maybe we can find some hack like local environment variable, rather than polluting serverless.yml etc..

  • thinking if starting multiple services together on local can use the same dynamodb local database, just placing multiple tables in it
  • default is to store data in memory, which gets destroyed each time we stop, if this becomes unmanageable or if we want to maintain state it is possible to save to file
  • all resource serverless.yml have below code, used only for DynamoDB local
  • (not sure) we do not migrate or seed here because we want to do that per service (eg in a script), and might have different seeds for different situations
custom:
 dynamodb:
 # If you only want to use DynamoDB Local in some stages, declare them here
   stages:
     - Dev
   start:
     port: 8000
     inMemory: true
     heapInitial: 200m #not sure correct setting, can leave off
     heapMax: 1g #not sure correct setting, can leave off
     migrate: true
     seed: true
   seed:
     default:
       sources:
         - table: ${self:service}${self:provider.stage}Config
           sources: [./__test__/dynamodb_seed_data/default/Config.json]

to install into a project, in resource dir: 

npm install --save-dev serverless-dynamodb-local

DynamoDB local web interface CRC32 errors

  • on one computer received this when using Mozilla only, in Chrome is works.
  • DynamoDB has some issues that might cause this, probably related to CORS, some browsers correctly follow CORS directives and some don't, there is a header Dynamo sends that states the CRC value, if it is striped by the browser this error does not occur.
  • Often happens in react apps
  • In those environments can turn off CRC checks
  • Also Dynamo zips larger response payments, but CRC is done incorrect, after/before zipping which does not match the clients CRC calc

Unit and Integration Tests for Local Development

  • Within our local environment we can have both unit and integration tests
  • Unit tests test 1 function only
  • Integration tests pass across multiple functions (basic functions, not lambda - although other lambdas in this service invoked sync might also work)
  • Unit tests might use mocks to bypass invoking other functions

Seed Database Data Shared by Local/Deployed Test

  • each service has its own seed data to inject into DynamoDB
  • place under test directory, group into seed sets (use default group to begin with)
  • then separated one file per tablename

Service Specific Unit/Integration Tests

  • in addition to flow tests pulled from Test Service allow for locally stored unit tests so developers can quickly check code
  • use same config structure as flow tests
  • separate unit and integration tests
  • use below file structure

Jest Test File/Directory Structure

  • at top level of project (eg inside /app dir) create a __test__ dir
  • under __test__ dir have integration and unit directories
  • under integration can use any logical dir structure, will likely mimic higher level dir src structure
  • under unit directory mimic the src directory structure for unit tests
  • under top level __test__ dir add dynamodb_seed_data dir
  • Jest by default runs all files in __test__ dir (maybe only top level), or with extensions .test.js or .spec.js
  • code that we want to run as testing code use .test.js extension

Working with AWS Mock (aws-sdk-mock)

  • Had a lot of trouble with AWS Mock associating itself with the wrong AWS instance (eg the instance in the test script, or the instance in the logic)
  • Will show as not able to find region, because trying to connect to actual AWS service when really we want it mocked locally
  • It would depend on whether the AWS service (eg AWS.Lambda) was created in global module scope in the test script, or within the function, if * inside the function was not so much of a problem, understand because it gets created after aws mock has attached itself to the AWS instance, but not if in global scope
  • I am not keen on creating the service inside the function because I guess it would be more resource intensive (creating a new service object each time the function is called)
  • Seemed to get around this by requiring aws-sdk before aws-sdk-mock, but still sometimes did not work
  • In middleware lambda.js test it would not connect, fixed by adding

AWSMock.setSDKInstance(AWS);

  • .. in the test script after requiring aws-sdk then aws-sdk-mock
  • Can try the same technique if other tests have trouble

Multiple aws.sdk modules

  • If start getting above issues, check to see there are not more than one aws.sdk modules included in node_modules directory
  • On one occasion we copied our middleware module into a projects node_module directory and also copied its node_modules directory which had a copy of aws.sdk in it
  • the middleware then used its own aws.sdk instance from its node_module directory instead of using the parent projects aws.sdk instance, causing the above issue

Process to Setup Local Environment for Tests

One Time Setup

Basic

  • clone repo’s
  • install dependencies in both resource and app dirs:
  • npm install

DynamoDB Local

  • install DynamoDB Local in one services resource dir, this only has to be done once then can be used by all services

this downloads dynamodb-local from aws and sets up 

sls dynamodb install

Per Session / Per Reboot

Start an empty instance of DynamoDB

  • can by in any services resource dir:
sls dynamodb start
  • this will also migrate and seed this service

DynamoDB (if working on multiple services)

  • Can seed multiple services into the local DynamoDB instance
  • resource dir of additional services:
sls dynamodb migrate
sls dynamodb seed

Multiple Dynamo Seed groups

  • if service has multiple seed groups, to only seed the default group:
sls dynamodb start --seed:default

Tests in Deployed Test Environment

  • Thinking we could have a separate service that holds all test configs (references locally to run tests)

Seed DynamoDB on AWS

  • If we want to seed remote DynamoDB tables with the same test seed data as dynamodb-local can use this function after deploy:

sls dynamodb seed --seed=default --online --region=us-east-1 --stage=Dev

  • --seed=default : default is the dynamodb local seed domain
  • must change region and stage according to deploy

Tests in Deployed Production Environment

Unit tests

  • tests that adjust state, eg database records, cannot be performed in production, but unit tests that do not mess with state could also be tested in production
  • mark or separate these in test config so could apply to production later, eg as part of CI/CD flow

Integration Tests

  • not sure yet, non-state adjusting tests could be run now and then
  • or could hook real workflows and check handling - difficult because how do we not expected results of live actions? Maybe just test that the flows process to completion..


Unused Notes

More Complex Offline AWS Environment

  • developers have a lot of trouble mimicking AWS offline, more complicated flows including DynamoDB streams etc might not be possible
  • also a simple setup from serverless.yml’s could be difficult if resources stretch across projects/files
  • localstack tries to emulate, but many have difficulty executing, is also a complicated setup and resource hog (apparently):

https://github.com/localstack/localstack https://medium.com/manomano-tech/using-serverless-framework-localstack-to-test-your-aws-applications-locally-17748ffe6755

  • localstack can integrate/get setup by servelerless.yml using:

https://github.com/localstack/serverless-localstack

  • not sure how to inject multiple projects into a single stack..

Thinking: If local queues etc fall apart or flow becomes too complicated:

  1. use mock return values to test each unit individually:

https://github.com/dwyl/aws-sdk-mock https://hackernoon.com/better-local-development-for-serverless-functions-b96b5a4cfa8f

  1. integration tests in live test stack on AWS, perhaps as part of CI flow

Old thoughts:

  • decided to stop using serverless offline (invoking tests by sending REST request to localhost API gateway which then triggered lambda) because:
  • cannot see a clean way to add mocks, which get setup in testing script.
  • we would need to use direct invoke anyway for isolated function tests that are not lambda handlers, so we already need to create structure for direct invoking, may as well use for everything
  • API Gateway/Lambda/DynamoDB offline appear to be well maintained so use these
  • Skip SNS/SQS offline modules, instead direct invoke lambda with requests that match SQS format, and mock outgoing requests

Start Request: Lambda triggered by API

Jest send REST request to local API server

Start Request: Lambda triggered by SQS

Jest direct invoke lambda

Start Request: Lambda triggered by direct invoke

Jest direct invoke lambda if lambda is designed to be a sync invoke, then it will likely return a value if designed to be async invoke then other outputs would be expected, eg messages sent to (mocked) queue, or Dynamo data updated. For Dynamo data updated from an async invoked lambda, when testing maybe we can invoke sync which will force Jest to wait for execution to complete before checking Dynamo results

Check Response: Lambda API Return Value

Jest receives response from the REST request If lambda being tested is designed to invoke async means the calling code does not wait for response. We can perhaps force Jest to invokes sync so Jest waits for execution to complete before checking results (eg in a Dynamo table)

Check Response: Lambda adjusts Dynamo data inside requested function

Jest checks database after API request returns I understand the lambda function should not return until all async operations are complete, otherwise lambda cleanup can dump anything remaining in call stack, so the data should be changed before Jest gets the response from REST request (an alternative we probably do not need to use is a Dynamo stream triggering Jest)

Check Response: Lambda sends message to SNS queue

Jest mocks SNS queue send message request from lambda returns success message gets received into Jest for testing

Check Response: Lambda direct invokes another Lambda within existing service

We could either mock the lambda.invoke call so the target lambda never gets invoked, and pass back expected response (if invoking sync). Also Jest can test the lambda.invoke request is as expected Or we could allow the invoke to happen, then test resulting outputs (from either lambda)(would be considered an integration test)

Check Response: Lambda direct invokes another Lambda outside existing service

probably mock the lambda.invoke call so the target lambda never gets invoked, does not need to be spun up if it is a sync invocation will likely expect a result, will need to build that into mock/test config

Local Modules

Serverless Offline

copied to Local test environment#Serverless Local

sets up a local api gateway and lambda functions adds environment variable IS_OFFLINE into Lambda functions

lambdas deployed to a local serverless offline environment are invoked by sending REST request to local API Gateway endpoint good for testing API Gateway config bad: cannot setup test environment (eg mocks) in the Jest code because is a separate environment to the serverless offline environment cannot test individual functions that are not API Gateway triggered

add to app serverless.yml below code only needed where functions and api gateway are deployed not needed in resource (if using sns/sqs offline they might need it) 

plugins:
  - serverless-offline

Serverless SNS offline

https://www.npmjs.com/package/serverless-offline-sns

  • documentation says cannot setup subscriptions triggering lambda function, although the example auto configures a lambda trigger from serverless.yml - not so important because we will probably always delivery to SQS queues
  • was unable to achieve SNS>SQS subscription, does not auto setup from serverless.yml, see:

https://github.com/mj1618/serverless-offline-sns/pull/88

  • tried to get around that by manually subscribing, which we would need to do anyway to mimic out Config db structure, manually subscribing, eg:
var AWS = require("aws-sdk");
...
var sns = new AWS.SNS({
   endpoint: "http://127.0.0.1:4002",
   region: "ap-southeast-1",
 });
 
 var sqs = new AWS.SQS({
   endpoint: "http://localhost:9324",
   region: "ap-southeast-1",
 });
 
 const listTopics = await sns.listTopics({}).promise(); // always shows empty even when SNS is functioning locally
 console.log("listTopics", listTopics);
 
 const queue = await sqs.createQueue({ QueueName: 'my-queue' }).promise();
 console.log("queue", queue)
 
 const subscription = await sns.subscribe({
   TopicArn: "arn:aws:sns:ap-southeast-1:123456789012:TestTopic",
   Protocol: 'sqs',
   Endpoint: queue.QueueUrl,
 }).promise();
 
 const listSubscriptions = await sns.listSubscriptions({}).promise();
 console.log("listSubscriptions", listSubscriptions);
 
 sns.publish({
   Message: '{"default": "hexxllo!"}',
   MessageStructure: "json",
   TopicArn: "arn:aws:sns:ap-southeast-1:123456789012:TestTopic",
 }).promise();

always creates a subscription regardless if SQS/SNS are valid can check subscriptions: aws sns --endpoint-url http://localhost:4002 list-subscriptions but no matter what TopicArn/Endpoint values I entered servers would always throw errors when trying to pass the message SNS> SQS, error suggested undefined sqs object, I assume unable to find the SQS queue stopped digging deeper because decided to use mocking with test seeds that check each step of workflow

add where deployed (resource): 

plugins:
 - serverless-offline-sns

I believe needs to be before serverless-offline in plugin list


had trouble where serverless-offline + serverless-offline-sns through error referring to already deployed SQS as event/trigger for lambda > project was using serverless-offline v3, serverless-offline-sns needed v5

add to serverless.yml: 

custom:
 serverless-offline-sns:
   port: 4002
   debug: true

Serverless SQS offline

https://www.npmjs.com/package/serverless-offline-sqs requires separate queue software installed queue, using ElasticMQ I believe requires serverless-offline in plugins to work

add to resource serverless.yml: 

 serverless-offline-sqs:
#   autoCreate: true                 # create queue if not exists
   apiVersion: '2012-11-05'
   endpoint: http://0.0.0.0:9324
   region: ap-southeast-1
   accessKeyId: root
   secretAccessKey: root
   skipCacheInvalidation: false

If have functions in app that serverless sets up with SQS events/triggers: also add above to app serverless.yml need to uncomment autoCreate: true requires serverless-offline-sqs higher version (v1 did not work, bumped to v3 worked)

If have SQS that subscribe to SNS within same project Currently SQS offline does not setup the SQS queue from serverless.yml Resources>AWS::SQS::Queue definition, it needs a function that is triggered by the SQS queue for it to be created That should be OK because should always have function in app that serverless sets up with SQS events/triggers (our SQS queues only exist to trigger a specific lambda) I thought would need to start app first in this situation, to create the SQS queue, then resource to create the SNS and make the subsription, but SNS->SQS subscriptions do not get auto-created by serverless-offline-sns at this time

ElasticMQ

used as the queue service for serverless-offline-sns must be installed, and started up each time before running serverless-offline https://github.com/softwaremill/elasticmq

Useful commands: aws sqs --endpoint-url http://localhost:9324 list-queues aws sqs --endpoint-url http://localhost:9324 send-message --queue-url http://localhost:9324/queue/TestQueue --message-body "MyFirstMessage" & aws sqs --endpoint-url http://localhost:9324 get-queue-attributes --queue-url http://localhost:9324/queue/TestQueue --attribute-names All aws sns --endpoint-url http://localhost:4002 list-subscriptions For executing node code, eg to play with subscriptions: node -e 'require("./subscribe_sqs_to_sns").main()'

example function definition that is triggered by SQS: 

 subscribed_function:
   handler: handler.subscribed_function
   description: subscribed_function
   events:
     - sqs:
         arn: arn:aws:sqs:ap-southeast-1:000000000000:TestQueue
         batchSize: 1

API Gateway Custom Authoriser

Currently seems to be skipped by serverless-offline because is set to a function not deployed in current service That is OK, because it would involve multi-service processing, locally we can use unit tests on the authorizer functions to check works. Do proper tests in deployed environment

Consider Scripts to start up Local Environment

Not sure want to happen in test code, or outside as a script If in test code less commands to start testing, but will take longer to fire-up/close-down each time run tests Some commands cannot be started in code? eg starting up serverless-offline/dynamo local? If we use commands might be more difficult to setup tests in CI process for now start up using commands info on starting serverless offline inside test script: https://dev.to/didil/serverless-testing-strategies-4g92

could create script/s (bash?) that startup the local services and inject database data if can make database data persist maybe split out data injection into a one off separate script, and have a combined script for first setup, then only run a start services script if using existing injected data script might have to branch off other terminals for long running commands like serverless offline, or dynamo, each has its own terminal

API Gateway ports

standardise ports for each local service’s apigateway so can operate/test on multiple services at one time no longer needed for inter-service communicated in local environment, so not that serious can configure in serverless.yml, or in batch that starts things up (prob serverless.yml for simplicity) was planning on setting port etc settings for serverless-offline type modules as command arguments to keep serverless.yml clean but standard seems to be to place them in serverless.yml, and we want to kind of fix them as standards anyway so placed them in the yml

Process to Setup Local Environment for Tests

One Time Setup

ElasticMQ for SQS queues

only needs to be done once, then all services can use download stand alone distribution: https://s3-eu-west-1.amazonaws.com/softwaremill-public/elasticmq-server-0.15.3.jar requires Java 8+

Per Session / Per Reboot

Shared

Startup ElasticMQ in directory where ElasticMQ package is saved: java -jar elasticmq-server-0.15.3.jar

For Each Service - Step 1: Resources

Start serverless offline (idea for starting multiple services, overkill for developing 1 service) in app directory for each service we standardize the ports for each service so matches our Config table seed data, can change/overwrite if needed but will need to update seed data in all services that connect to endpoints examples: start from parent dir that holds all local project repos needs gnome/bash $SHELL keeps terminal open after server stops gnome-terminal -e "bash -c 'cd ./permission-handler/app; serverless offline --port=29320; $SHELL'" gnome-terminal -e "bash -c 'cd ./account-limit/app; serverless offline --port=29321; $SHELL'"

Start SNS queues

in resource directory for all projects that have SNS serverless offline-sns start


References

testing async functions (uses Mocha but similar to Jest): https://medium.com/serverlessguru/how-to-unit-test-with-nodejs-76967019ba56

package for mocking services: https://github.com/dwyl/aws-sdk-mock

standardize ref names for AWS and local deployment in serverless.yml: https://www.jeremydaly.com/developing-serverless-applications-locally-with-the-serverless-cloudside-plugin/

designing lambda functions to be testable: https://claudiajs.com/tutorials/designing-testable-lambdas.html

Retrieved from "https://izara.io/wiki/index.php?title=2019-01-01_-_Mocking_AWS_resources_in_local_test_environment&oldid=451"