Service - User Resource Use

Overview

Records resource use allocated to each user, this can be used to bill users according to their resource use, eg after free limits have been exceeded.

Repository

https://bitbucket.org/izara-core-user-accounts/izara-core-user-accounts-user-resource-use

DynamoDB tables

Standard Config Table Per Service

Configuration tags

{
	configKey: "UserAccountBalanceServiceName"
	configTag: "UserAccountBalanceServiceName"
	configValue: xxx // eg: "AccBalance"
}

{
	configKey: "resourceTag"
	configTag: ".." // eg LambdaExecutionTime|DynamoDbRead|DynamoDbWrite|Neo4j|..
	configValue: {
		"freeLimit": xx
	}
}

ResourceUse

{
	userIdResource: xxx
	useTime: xxx
	details: xxx
	usedUnits: xxx
}

• partition key: userIdResource
• sort key: useTime
• userIdResource: userId + "_" + "resourceTag"
• useTime: {timestamp of use}_{uniqueRequestId}, add uniqueRequestId to ensure each record is unique
• details: an object with additional information about the resource use, eg what Lambda function, query, etc..
• resourceTag: Lambda/Dynamo/Neo4j/etc..
• usedUnits: Quantity of used per resourceTag

UserTransactionRecord

{
	userId: xxx
	firstTransactionTime: xxx
	partitionId: xxx
}

• partition key: userId
• no sort key
• partitionId: maybe a fixed range of partition keys, eg 10, that breaks up records a little
• try to generate partitionId somehow from userId in consistent way so standard partitionId for a single user, otherwise querying existing record for a user cannot be done
• firstTransactionTime: useTime timestamp for the earliest resource use that has not had a transaction created yet
• we will be querying based on firstTransactionTime being older than transaction period, partition key is not really needed
• have GSI:
  • partition key: partitionId
  • sort key: firstTransactionTime
  • project: userId
  • used to find which users need to process transactions, eventual consistency is ok

Reporting use in services

• some actions will be billed the signed in user, eg work on shared resources like complexfilters, others according to targetuserid, this will need to be set on a per lambda basis
• middleware (or RecordHandler?) will record in iz_Context eg which user is being charged and other detail like start time of Lambda execution
• libraries like dynamoDb will send messages for it's use
• async batch processing might be difficult to record which time is for which record, maybe not worry about it, do the best we can without too much resource use
• services will need Config table entry for UserResourceUse serviceName

Creating Account Balance transaction

• Each user will have a transaction created periodically for the amount of resources they use
• Want to only check for users that had some transactions
• Can incorporate free limits during each period, only create transaction when exceed
• A period begins from the time of the earliest resource use since last transaction calculation

How to know which resourceUse records to process

• when a resource use is recorded it checks if no record exists in UserTransactionRecord for this user, if no it conditionally creates a record with the current resourceUse timestamp
• conditional insert will check if either no record still exists (create) or record exists and firstTransactionTime later than current resourceUse timestamp (update/overwrite)
• if existing record exists, check firstTransactionTime is after current resourceUse timestamp, if yes then update the record (maybe conditional to catch race condition when resourceUse transaction processed this user, but perhaps not need that level of accuracy)
• consider read before put rather than direct conditional write query, might be cheaper Dynamo capacity (reads are 1/5 price of writes, conditional update always costs write capacity)
• to find users that need to calculate new transaction iterate through partitionIds and filter firstTransactionTime for records that are older than transaction period

Calculating transaction value

• each resourceTag will have a multiplier that takes the total resourceUse for that resourceTag since the recorded firstTransactionTime, and multiplies it to achieve the transaction amount in USD
• each resourceTag can have a free limit, only the amount exceeding this limit during the given period gets charged
• free limits can be stored in Config table per resourceTag, but in future could adjust on a per user basis, eg from user roles

Ideas

• resourceUse table could have a TTL to clean up old values, perhaps triggering a DynamoDB stream that stores the records in cheaper location, eg S3

Working documents

User Resource Use