"Excluding Development Dependencies" Takes Forever

aws
lambda

#1

Hi,

I’m not sure what to do! I have about 20 functions. I do sls deploy and it takes about 15 minutes at this point. The CPU is spinning on something when I look at htop.

The resulting packages are very small (all < 1MB).

Here’s a snapshot of .serverless after all the artifacts have been generated.

-rw-r--r-- 1 dawilcox dawilcox 356K Feb  3 13:49 adAccountDetails.zip
-rw-r--r-- 1 dawilcox dawilcox  72K Feb  3 13:49 addCustomer.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:42 addFundraiserMedia.zip
-rw-r--r-- 1 dawilcox dawilcox 468K Feb  3 13:43 aliceCreateAdFinalize.zip
-rw-r--r-- 1 dawilcox dawilcox 468K Feb  3 13:39 aliceCreateAdFulfillment.zip
-rw-r--r-- 1 dawilcox dawilcox 468K Feb  3 13:49 aliceCreateAdValidator.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:49 alicePostPicture.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:49 alicePostValidation.zip
-rw-r--r-- 1 dawilcox dawilcox 336K Feb  3 13:49 aliceScheduleDemo.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:39 aliceStatement.zip
-rw-r--r-- 1 dawilcox dawilcox 337K Feb  3 13:49 authorizerFunc.zip
-rw-r--r-- 1 dawilcox dawilcox  483 Feb  3 13:30 cloudformation-template-create-stack.json
-rw-r--r-- 1 dawilcox dawilcox 138K Feb  3 13:49 cloudformation-template-update-stack.json
-rw-r--r-- 1 dawilcox dawilcox 448K Feb  3 13:49 createAd.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 createFundraiser.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 createUser.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 getUserInfoByEmail.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 getUserInfo.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:43 handleAliceSynchronous.zip
-rw-r--r-- 1 dawilcox dawilcox 571K Feb  3 13:39 handleAlice.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 instagramConnect.zip
-rw-r--r-- 1 dawilcox dawilcox 348K Feb  3 13:39 postFacebookMessage.zip
-rw-r--r-- 1 dawilcox dawilcox 348K Feb  3 13:40 postFacebookPicture.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:43 removeFundraiserMedia.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:43 removeFundraiser.zip
-rw-r--r-- 1 dawilcox dawilcox 193K Feb  3 13:49 serverless-state.json
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:49 setUserAttributeValue.zip
-rw-r--r-- 1 dawilcox dawilcox 347K Feb  3 13:43 updateFundraiser.zip

After the artifacts are generated, they get uploaded and deployed very quickly. But, generating them is very slow.

I have just a few things in my package.json:
{

“dependencies”: {
“request”: “^2.83.0”
},
“devDependencies”: {
“chai”: “^4.1.2”,
“mocha”: “^4.0.1”,
“serverless-dynamodb-local”: “^0.2.26”,
“serverless-offline”: “^3.16.0”,
“serverless-plugin-optimize”: “^3.0.4-rc.1”
},

}

I put everything in the lib directory under serverless, then deploy functions individually.
dawilcox@oso:~/back$ ls -l lib/
total 64
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 28 19:18 adAccount
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 31 21:46 alice
drwxr-xr-x 2 dawilcox dawilcox 4096 Jan 30 23:13 aliceCreateFbAd
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 5 22:23 alice_demo
drwxr-xr-x 2 dawilcox dawilcox 4096 Jan 27 08:15 aliceFundraisers
drwxr-xr-x 3 dawilcox dawilcox 4096 Feb 2 20:31 authorization
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 5 22:23 customers
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 30 23:16 fb_create_ad
-rw-r–r-- 1 dawilcox dawilcox 5480 Jan 31 21:57 fb.js
drwxr-xr-x 70 dawilcox dawilcox 4096 Dec 24 20:11 node_modules
-rw-r–r-- 1 dawilcox dawilcox 344 Jan 4 23:27 package.json
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 25 20:10 post
drwxr-xr-x 3 dawilcox dawilcox 4096 Jan 28 20:48 post_picture
drwxr-xr-x 2 dawilcox dawilcox 4096 Jan 31 20:52 shared
drwxr-xr-x 3 dawilcox dawilcox 4096 Feb 2 22:01 users

Each of those directories has its own package.json that has the libraries for each one.

My serverless.yml file has:

package:
  exclude:
          #- node_modules/**
    - test/**
    - test-events/**
    - .git/**
    - scripts/**
    - config/**
    #- **/node_modules/aws-sdk/**
  include:
    - lib/node_modules/
    - lib/fb.js
    - lib/shared/
  individually: true

Then, here’s an example function:
functions:
createUser:
handler: lib/users/users.create
events:
- http:
path: user/create
method: post
cors: true
authorizer: authorizerFunc
include:
- lib/users/

Do you have any clue why sls deploy might take 15 minutes for just 20 functions? I dread running it at this point because I know I’m in for a trip every time.


Not excluding dev deps
#2

Anything listed in package.json as a dev dependancies should be automatically excluded.

It sounds like one of your include/exclude patterns is glob matching much more than it needs.

Do you need to explicitly define the include/exclude because of your project structure?


#3

Well. Yes. You can see from above that I exclude .git (doesn’t that make sense)? Is it not normal to exclude .git? Maybe I’m misunderstanding the normal method for doing this type of thing. Do you have documentation for what is by default included or excluded in a serverless deployment package? Also, I have a lot of things in my node_modules like serverless offline. My impression from awhile ago was that serverless packages up everything, so you have to specifically exclude things that don’t make sense.


#4

anything listed in gitignore and package.json devDependancies get excluded by default (since idk like 7 versions ago).

.git folders shouldn’t be included in the deploy zip of your service


#5

OK. And how do libraries work when doing build individually? How do I make sure that only the libraries for a given project are deployed? So, if a given endpoint needs library A, but not library B, then what’s the best way to manage that?

So far, I’ve had a package.json and node_modules for every one of my source directories that gets deployed individually. Maybe that’s trying to manage too much, though.


#6

i also have the same problem:
here is my server.yml:

service: myapi

provider:
  name: kubeless
  runtime: python2.7
  environment:
      DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}

plugins:
  - serverless-kubeless

functions:
  get-post:
    handler: get_post.get_post

also if i delete the function with kubeless and deploy after (like below),the commnad doesn’t stuck:
kubeless function delete get-post
sls deploy

any idea why it is taking so long?


#7

This pr might fix the issue. Would you be able to test?

Cheers!