"Excluding Development Dependencies" Takes Forever

Hi,

I’m not sure what to do! I have about 20 functions. I do sls deploy and it takes about 15 minutes at this point. The CPU is spinning on something when I look at htop.

The resulting packages are very small (all < 1MB).

Here’s a snapshot of .serverless after all the artifacts have been generated.

-rw-r--r-- 1 dawilcox dawilcox 356K Feb  3 13:49 adAccountDetails.zip
-rw-r--r-- 1 dawilcox dawilcox  72K Feb  3 13:49 addCustomer.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:42 addFundraiserMedia.zip
-rw-r--r-- 1 dawilcox dawilcox 468K Feb  3 13:43 aliceCreateAdFinalize.zip
-rw-r--r-- 1 dawilcox dawilcox 468K Feb  3 13:39 aliceCreateAdFulfillment.zip
-rw-r--r-- 1 dawilcox dawilcox 468K Feb  3 13:49 aliceCreateAdValidator.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:49 alicePostPicture.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:49 alicePostValidation.zip
-rw-r--r-- 1 dawilcox dawilcox 336K Feb  3 13:49 aliceScheduleDemo.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:39 aliceStatement.zip
-rw-r--r-- 1 dawilcox dawilcox 337K Feb  3 13:49 authorizerFunc.zip
-rw-r--r-- 1 dawilcox dawilcox  483 Feb  3 13:30 cloudformation-template-create-stack.json
-rw-r--r-- 1 dawilcox dawilcox 138K Feb  3 13:49 cloudformation-template-update-stack.json
-rw-r--r-- 1 dawilcox dawilcox 448K Feb  3 13:49 createAd.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 createFundraiser.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 createUser.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 getUserInfoByEmail.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 getUserInfo.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:43 handleAliceSynchronous.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:39 handleAlice.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 instagramConnect.zip
-rw-r--r-- 1 dawilcox dawilcox 348K Feb  3 13:39 postFacebookMessage.zip
-rw-r--r-- 1 dawilcox dawilcox 348K Feb  3 13:40 postFacebookPicture.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:43 removeFundraiserMedia.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:43 removeFundraiser.zip
-rw-r--r-- 1 dawilcox dawilcox 193K Feb  3 13:49 serverless-state.json
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 setUserAttributeValue.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:43 updateFundraiser.zip

After the artifacts are generated, they get uploaded and deployed very quickly. But, generating them is very slow.

I have just a few things in my package.json:
{

“dependencies”: {
“request”: “^2.83.0”
},
“devDependencies”: {
“chai”: “^4.1.2”,
“mocha”: “^4.0.1”,
“serverless-dynamodb-local”: “^0.2.26”,
“serverless-offline”: “^3.16.0”,
“serverless-plugin-optimize”: “^3.0.4-rc.1”
},

}

I put everything in the lib directory under serverless, then deploy functions individually.
dawilcox@oso:~/back$ ls -l lib/
total 64
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 28 19:18 adAccount
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 31 21:46 alice
drwxr-xr-x 2 dawilcox dawilcox 4096 Jan 30 23:13 aliceCreateFbAd
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 5 22:23 alice_demo
drwxr-xr-x 2 dawilcox dawilcox 4096 Jan 27 08:15 aliceFundraisers
drwxr-xr-x 3 dawilcox dawilcox 4096 Feb 2 20:31 authorization
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 5 22:23 customers
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 30 23:16 fb_create_ad
-rw-r–r-- 1 dawilcox dawilcox 5480 Jan 31 21:57 fb.js
drwxr-xr-x 70 dawilcox dawilcox 4096 Dec 24 20:11 node_modules
-rw-r–r-- 1 dawilcox dawilcox 344 Jan 4 23:27 package.json
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 25 20:10 post
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 28 20:48 post_picture
drwxr-xr-x 2 dawilcox dawilcox 4096 Jan 31 20:52 shared
drwxr-xr-x 3 dawilcox dawilcox 4096 Feb 2 22:01 users

Each of those directories has its own package.json that has the libraries for each one.

My serverless.yml file has:

package:
  exclude:
          #- node_modules/**
    - test/**
    - test-events/**
    - .git/**
    - scripts/**
    - config/**
    #- **/node_modules/aws-sdk/**
  include:
    - lib/node_modules/
    - lib/fb.js
    - lib/shared/
  individually: true

Then, here’s an example function:
functions:
createUser:
handler: lib/users/users.create
events:
- http:
path: user/create
method: post
cors: true
authorizer: authorizerFunc
include:
- lib/users/

Do you have any clue why sls deploy might take 15 minutes for just 20 functions? I dread running it at this point because I know I’m in for a trip every time.

Anything listed in package.json as a dev dependancies should be automatically excluded.

It sounds like one of your include/exclude patterns is glob matching much more than it needs.

Do you need to explicitly define the include/exclude because of your project structure?

Well. Yes. You can see from above that I exclude .git (doesn’t that make sense)? Is it not normal to exclude .git? Maybe I’m misunderstanding the normal method for doing this type of thing. Do you have documentation for what is by default included or excluded in a serverless deployment package? Also, I have a lot of things in my node_modules like serverless offline. My impression from awhile ago was that serverless packages up everything, so you have to specifically exclude things that don’t make sense.

anything listed in gitignore and package.json devDependancies get excluded by default (since idk like 7 versions ago).

.git folders shouldn’t be included in the deploy zip of your service

OK. And how do libraries work when doing build individually? How do I make sure that only the libraries for a given project are deployed? So, if a given endpoint needs library A, but not library B, then what’s the best way to manage that?

So far, I’ve had a package.json and node_modules for every one of my source directories that gets deployed individually. Maybe that’s trying to manage too much, though.

i also have the same problem:
here is my server.yml:

service: myapi

provider:
  name: kubeless
  runtime: python2.7
  environment:
      DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}

plugins:
  - serverless-kubeless

functions:
  get-post:
    handler: get_post.get_post

also if i delete the function with kubeless and deploy after (like below),the commnad doesn’t stuck:
kubeless function delete get-post
sls deploy

any idea why it is taking so long?

This pr might fix the issue. Would you be able to test?

Cheers!

Late to the party, but this recent update helped a lot! Thanks!!!

1 Like

Doesn’t look like this has been fixed yet. Excluding development dependencies… still takes forever for me.

4 Likes

The same problem for me, take around 7-8 minutes

I was also having the same problem. The below seemed to work for me

package:
  individually: true
  excludeDevDependencies: false
  exclude:
   - node_modules/**

Note that I am using layers to hold my node module dependency so the above is just for my functions.

3 Likes

I was having the same problem. It turned out that for various reasons, the project structure had about 250,000 files in it (links in node_modules, …). In serverless.yml, we’d added an exclude glob that would get rid of about 244,000 of them. The slowdown happened when packaging was processing that exclude glob.

We fixed it by changing serverless.yml from

package:
  exclude:
    - node_modules/@modulenamespace/*/node_modules/**

to

package:
  include:
    - "!node_modules/@modulenamespace/*/node_modules/**"
2 Likes

Changing from an exclude to a !include dropped my deploy to like 10% of original duration. Shocked this simple change was so effective, thanks!