Lambda with RDS using VPC works slow

Hi everyone,

I am trying to use Lambdas with RDS and what I am discovering right now that lambdas works too slow.
I have one function that use DynamoDB and runs around 500ms, but another function that uses RDS runs 12sec (cold start) and ~3sec on repeat…
First I was connecting without VPC, using Publicly AccessibleWhether the instance is publicly accessible and it shows the same results.
My lambda runs SELECT * FROM users where table has around 10 rows in 3 seconds…
I want to know if I am doing smth wrong or it’s expected behaviour?

Thanks in advance.

Are you using Sequelize?

On the first line after your functions module definition, add the following line:

context.callbackWaitsForEmptyEventLoop = false;

3 Likes

Did that help at all?

Yes! i am using Sequelize

Did my suggestion help you?

Yes, thanks. It works like a charm.

I’m using the pg module directly, and this helped me to shave off exactly 10 seconds of every request :joy:

The postgres module is, by default, pooling up connections, and doesn’t let them go until 10 seconds of idle time. Therefor the requests didn’t finish until the pool timed out the connection and closed it.

What’s very interesting is that Amazon themselves encourage this very behaviour:

Take advantage of container re-use to improve the performance of your function. Make sure any externalized configuration or dependencies that your code retrieves are stored and referenced locally after initial execution. Limit the re-initialization of variables/objects on every invocation. Instead use static initialization/constructor, global/static variables and singletons. Keep alive and reuse connections (HTTP, database, etc.) that were established during a previous invocation.

ref: Best practices for working with AWS Lambda functions - AWS Lambda

Oh my god it’s taken me 2 days to figure out why my RDS responses time out.

Just be aware that using that setting means that if you have a callback outside of an asynchronous operation (such as a Promise or anonymous function) that the code executing within those asynchronous blocks may not complete before the lambda execution ends.

Hmm, it is inside a promise (using pg-promise). Is there a better way?

If your function callback is within a promise that’s fine. Just be aware if you initiate a Promise it gets added to the event loop and if outside that promise, after you initiated it, you run your function callback that promise may or may not finish executing.

Also, in addition to

context.callbackWaitsForEmptyEventLoop = false;

If your Lambdas and RDS are on different VPCs or different AZs, that’ll be a problem too. Make sure they’re geographically close.

Turn on AWS X-Ray and you can get a detailed timing breakdown of your lambda function. In my case the the source of the slow down was not at all where I expected it to be.

Also note that X-Ray can cover much more than just lambda. You can set it up to instrument your entire AWS deployment.

1 Like

Thanks — that’s really helpful, wish I’d known about that earlier! :smiley: