I am running a cron job every 12 hours on aws lambda to do some processing on orders that have come through. It gets the orders for the past 12 hours, and then sends out a questionnaire. I’ll be processing up to 1000 orders on each invocation, and it may become larger. Right now, it’s all processed through one lambda function. Each order has a couple of api calls that it has to make too, in order to obtain more information about the order, and then to send the questionnaire.
My question is, what is the best practice for scaling, or how would other people handle it? Here are some options I have thought of.
- I know lambdas have a 15 minute max execution. I haven’t hit that, and probably won’t for a while. It could just keep scaling until that is hit.
- Have 1 lambda to obtain all of the orders, and then have that one invoke another lambda for execution of each order.
- Set the cron to run more often, for example once an hour as opposed to very 12 hours.
It’s working currently, but I’m trying to learn more about best practices and what other options might be.