"EMFILE: too many open files error" while CI/CD deploy

Serverless Framework
,
1

Hello. I have an organization with multiple nodejs lambda services. Every service have different Github repository and Ci/Cd integration. They works well until yesterday.
Yesterday, after i commited some code to one of my services, Ci/Cd deploy throwed an error. Here is my deploy logs ;

Build step: serverless deploy --stage dev --region us-east-1 --force --org myorg --app myapp --debug
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
            Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
            More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: User Configuration warning: Cannot resolve local config file.
Error: EMFILE: too many open files, open '/My_Service/.serverlessrc'
Serverless: User Configuration warning: Cannot resolve global config file: /root/.serverlessrc 
Error: EMFILE: too many open files, open '/root/.serverlessrc'
Serverless: Your previous global config was renamed to /root/.serverlessrc.bak for debugging. Default global config will be recreated under /root/.serverlessrc.
 
  Serverless Error ---------------------------------------
 
  Cannot read file node_modules/string-width/node_modules/strip-ansi/index.js due to: EMFILE: too many open files, open '/My_Service/node_modules/string-width/node_modules/strip-ansi/index.js'
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              12.19.0
     Framework Version:         2.28.3
     Plugin Version:            4.4.3
     SDK Version:               2.3.2
     Components Version:        3.7.1
 
Build step failed: serverless deploy --stage dev --region us-east-1 --force --org myorganization --app myapp --debug

That file always changes to different file. And there are two points.

  • I can upload my code from my local with sls deploy. It works well. No problem. So i think there is no problem with my yaml file. But if anyone needs that, i can share.
  • My other services gives same error. All Ci/Cd integrasions throws same error.

I tried to clear or delete my CloudFormation Snacks and buckets. Thats not worked. Anyone help me? Im so confused.

Serverless deploy works in local, but not within GitLab CICD. The erorr message seems empty
2

Having the same issue. Started this morning. @atakansavas Can you share your serverless.yaml?

3

Of course. I use nodejs with fastify.
Here is my file;

provider:
  httpApi:
    metrics: true
  name: aws
  runtime: nodejs12.x
  region: us-east-1
  apiGateway:
    shouldStartNameWithService: true
  vpc:
    securityGroupIds:
      - sg-***
    subnetIds:
      - subnet-***

plugins:
  - serverless-offline
  - serverless-apigw-binary

functions:
  app:
    handler: lambda.prepare
    events:
      - http: ANY /
      - http: 'ANY /{proxy+}'

custom:
  apigwBinary:
    types:
      - '*/*'
4

Hi again. Finally i can fix the problem. I dont know why, but when i try to deploy my lambda project to cloud, error says : my npm packages size is too big. And i decided to take a look again my package.json. Here is my solution steps;

  • Make sure which dependencies you need. Make you sure you use devDependencies.
  • individually: true flag in package helps me too.

Also, you can use serverless-bundle package on npm for webpack.

5

A fix for the issue related to EMFILE is being deployed later today.

1 Like
6

And could it be solved?

7

The fix was deployed last week so you should be able to update the framework and all is well again

8

How have you fixed for the issue related to EMFILE?

9

I just started getting this same issue last week. I upgraded to use the latest AWS sdks which got rid of the problem for a few days, but it came back again. I installed serverless-plugin-include-dependencies which seems to help deploy in our CI/CD (circleci) but Iā€™m not able to deploy locally and it feels like itā€™s just a problem waiting to rear its head in the near future.

Serverless: Packaging serviceā€¦

Serverless Error ----------------------------------------

Cannot read file node_modules/uuid/dist/esm-node/nil.js due to: EMFILE: too many open files, open ā€˜/Users/liron/Dev/SourceIP/sourceip/api/node_modules/uuid/dist/esm-node/nil.jsā€™

Your Environment Information ---------------------------
Operating System: darwin
Node Version: 12.22.8
Framework Version: 2.72.2
Plugin Version: 5.5.4
SDK Version: 4.3.0
Components Version: 3.18.2

11

Hello, I got the EMFILE error when launching the serverless ā€œdeployā€ script. I used the graceful-fs package like this to solve the issue :

  1. install graceful-fs: npm install graceful-fs
  2. go to file: node_modules/serverless/lib/plugins/package/lib/zip-service.js
  3. replace this line :
const fs = BbPromise.promisifyAll(require('fs'));

by this:

var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)
const fs = BbPromise.promisifyAll(realFs);
1 Like
12

Thankyou, this workaround solved my package and deploy failures with Serverless 3.39.0 on windows 11.

13

This is a great suggestion, but unfortunately Github Actions (GHA) pulls the Serverless from NPM so we canā€™t really do thatā€¦ :frowning:

We started getting this problem in a few repos, then we bumped up the GHA runner to a beefier server and it ran for a few months, but now weā€™re back to getting this ā€œEMFILE; too many filesā€ again regardless how many cores we throw at it as Github has a fixed ulimit on their runners which is the same regardless of the number of coresā€¦

Any other fix or suggestions in sight?