Methods of throttling DynamoDB writes

Serverless Architectures
1

I am looking to store data from a variety of sources into a DynamoDB table on a daily basis.

Currently, I have one lambda function that is triggered by a daily cronjob, which posts to a SNS topic with the information for every data source. These SNS posts then trigger individual lambda functions for each data source that pull data and stores it in a single DynamoDB table.

The problem is that all the individual lambda functions for each data source are attempting to write into the DynamoDB table at the same time, so I am vastly exceeding the allocated write throughput for my table. Is there a way to add a slight delay between the writes of these individual lambda functions? It looks like SQS may offer a solution to this scenario, but I do not believe it is a currently supported trigger of lambda functions?

2

You’re correct that SQS isn’t currently a trigger for Lambda.

Have you considering using up a data pipeline instead? http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part1.html

3

I haven’t used them (yet), but this sounds like something you could do with Step Functions - it has a wait condition.

Now that SFs are in ap-southeast-2 I might get a chance to use them!