Reusing Database Connections (node.js)

Serverless Framework
,
1

So I’m new serverless and for the most part super stoked. But having an issue with multiple db connections. I’m using serverless with node.js, typescript, mongodb.

The issue I’m having is a grip of database connections. I’ve searched everywhere and the consensus seems to have your database connection outside the handler, which I do but it seems to not work. I made connectToDB function that looks like this:

import * as mongoose from "mongoose";

mongoose.Promise = global.Promise;
let isConnected: boolean = false;
export const connectToDatabase = () => {
    if (isConnected) {
        console.log('using existing database connection');
        return Promise.resolve();
    } else {
        console.log('using new database connection', { isConnected });
        return mongoose.connect(process.env.DB_CONNECT)
            .then(db => {
                isConnected = true;
            });
    }
};

and I call it inside my handler like:

import { connectToDatabase } from '../../db';
export const somehandler: Handler = (event: APIGatewayEvent, context: Context, cb: Callback) => {
          context.callbackWaitsForEmptyEventLoop = false;
          connectToDatabase()
               .then(()=>{
                    model.findOne....
               })
}

For the life of me I can’t get it to reuse the connection. I just end up at the end of testing like… 45 to 100 connections that never seem to disconnect until I shut down local serverless offline.

Any direction would be much appreciated.

Thanks,

Chris

2

You need to connect to the database outside of your handler

3

Yea I’ve tried that and nothing seems to work. I created just an if statement outside the handler to call the db.connect, that didn’t work.

I’ve called the connectoDatabase function outside the handler and had a separate function the handler calls that hits the db inside of a try catch, that didn’t seem to work…

I don’t know what I’m doing wrong.

chris

4

I am using DynamoDB for my projects, so I cannot be of much help unfortunately. Not sure if you saw this blog post from Mongo:
https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs

5

@ahansson89 Thanks for giving it a try. Yea I looked over that post and quite a few others, and have tried everyone of them. I don’t know, maybe it has something to do with using local serverless-offline. I figure it out eventually.

Thanks,

Chris

6

Interesting, just as a reference: This works for me in serverless-offline:

const mongoose = require('mongoose');
const bluebird = require('bluebird');
mongoose.Promise = bluebird;
mongoose.Promise = global.Promise;

// Only reconnect if needed. State is saved and outlives a handler invocation 
let isConnected;

const connectToDatabase = () => {
  if (isConnected) {
    console.log('Re-using existing database connection');
    return Promise.resolve();
  }

  console.log('Creating new database connection');
  return mongoose.connect(process.env.MONGODB_URL)
    .then(db => {
      isConnected = db.connections[0].readyState;
    });
};

module.exports = connectToDatabase;

At first glance this only difference is that I save db.connections[0].readyState instead of hard setting it to true.

7

I’m having the same problem, and i’m certain it’s because of serverless offline.

Do you figure out a way?

Thanks

8

This is only an issue with serverless-offline and is caused by the way it does cache invalidation.

If you want a quick fix you can do sls offline start --skipCacheInvalidation which will start (and kill) a node process for every request. This slows things down but fixes the issue of database connections staying open. See this PR for details: https://github.com/dherault/serverless-offline/pull/368

3 Likes
9

Do you ever need to disconnect from Mongo in your lambda? Or do you just let the connection linger?

10

Hi ojongerius,

can you explain why you choose the first db

      isConnected = db.connections[0].readyState;

thanks

11

I had the same issue and you saved me so much frustration! The --skipCacheInvalidation flag did the trick and saved my local MongoDB from crashing from too many open documents. THANK YOU!!

12

This works EXCEPT it stops serverless-offline from auto-reloading on code changes. Webpack sees the change and recompiles (using serverless-webpack as well), but serverless-offline doesn’t swap code in require (see https://github.com/serverless-heaven/serverless-webpack/issues/352).

The thing that serverless-offline is not doing that aws lambda does is share a container for all function invocations. In AWS, it runs the code outside the handler when it creates the container, then invokes the function multiple times from the same container (until somewhere between 5 and 30 minutes, at which point it kills the container and creates a new one on the next invocation). This is why declaring the db connection outside the function makes it reusable. However, serverless-offline doesn’t seem to do this unless ---skipCacheInvalidation is set.

It would be nice if we could get this shared-container behavior in serverless-offline instead of creating a brand new one on every invocation.

1 Like
13

In case anyone lands here and is looking for a solution, you can run sls offline start --allowCache that should keep your mongodb connection (and any other variables defined outside the handler) live outside the handler.

1 Like
14

thanks man, it worked for me.

15

You’ve called the connectoDatabase function outside the handler and had a separate function the handler calls that hits the db inside of a try catch, that didn’t seem to work…

snaptube vidmate