I am going nuts, I have used Serverless successfully for months and now with a new project I can’t get it to exclude my Dev Dependencies. I have a test function foobar.js:
One thing that I noticed before I started packaging individually is that the size of the deployment package increased every time I would add a new function. So, node_modules was being included for every one of my N functions.
@DavidWells I looked at the related thread … seems that i was closed based on a PR that was incorporated in July. I’m currently on the latest build of serverless (1.26.0) so should be operating with the benefit of this PR. One thing I may have not fully digested was a suggestion to have a nodejs-package-json parameter … was this implemented (didn’t see it in PR commentary just as suggestion in the issue)? I have been trying to rely on include and exclude directives.
One other strategy I’ve tried – without success – is to exclude node_modules/** at the global package scope but then just including specific node_modules needed in the include blocks of individual functions. For example:
package:
individually: true
excludeDevDependencies: true
browser: false
exclude:
- node_modules/**
- src/**
- test/**
custom:
stage: "${opt:stage, self:provider.stage}"
region: "${opt:region, self:provider.region}"
provider:
name: aws
runtime: nodejs6.10
profile: rt
stage: dev
region: us-east-1
functions:
stateInfo:
package:
include:
- node_modules/axios/**
- node_modules/cheerio/**
- node_modules/entities/**
- node_modules/firemock/**
- node_modules/firemodel/**
environment: ${file(env.yml):${self:custom.stage}}
description: Get meta information on US States
handler: lib/state-info.handler
timeout: 5
memorySize: 1536
Doing this, however, seems to exclude all node_modules and then ignore the includes at the function level.
A bit desperate folks. I have lost weeks on this. I even tried to revert to rollup (for tree shaking) but that failed for a variety of reasons. If I can’t solve this today/tomorrow I’ll have to move away from serverless which would make me very sad as it’s a great framework but in this state it just isn’t usable.
This chatter doesn’t scale on a community forum, don’t expect an account manager to drop everything to make sure you keep using the product.
How is this blocking you? Personally the only benefit of a small package size for me is when I need to deploy from my home, which has limited upload bandwidth.
You haven’t shared your package.json, which begs the question, do you have any packages in the devDependecies section?
My suggestions:
Try multiple versions of the framework, shouldn’t too long to test the releases from 1.19 to 1.26. I say 1.19 because the last bug I had with omitting devDependencies was fixed there.
Serverless webpack, it bundles everything into a single .js file.
Wow, that was an aggressive response. I have lost a month of productivity researching bundling solutions (webpack, rollup, browserify) that will work around this problem and so my response … which initially garnished a response … was in hindsight entirely reasonable. I mean, we are talking about a show-stopper problem after all.
I have a 20 line function which uses Axios and yet it’s too big to be published as a single function? When I deploy the whole thing it goes up but ends up being 78mb. Webpack, rollup, and browserify all have problems working with google’s Firebase and I have not yet found a way to work around that so the obvious bundling solution doesn’t work for me either (i’m guessing that others may have worked around this problem via this solution).
Also, the ability to globally exclude all of node_modules and then at a function level bring back packages doesn’t seem to work … that was another hope. In the end I wrote a JS file to analysize my dependency tree and tease out the devDeps and all the modules which the devDeps were bringing in (aka, recreating the wheel that serverless apparently developed) and now I have a list of 100’s of packages that I’m excluding. That gets me back down to 47mb (still way too big but working).
Comparison: I have one function which has no npm dependencies so for it I just exclude every package … this uploads to a normal filesize of 5.4mb
Look, we’re a community. Slapping someone in your community is not a good thing to do. I think an apology from you @sime would be a good gesture. You assumed I was an idiot. I am not.
Note: the documentation points to a feature which allows you to point to a JSON config file for the full list of exclusions but it doesn’t specify the format of that JSON file. I had hoped it would just be a JSON array of directories to exclude but that doesn’t work so my serverless.yaml file has to contain the hundreds of exclusions (322 exclusions).
Can you share the serverless.yml or add me (github username DavidWells) to the repo.
I personally haven’t seen any issues with dev dependancies not being excluded from any of my services. So it sounds like something specific to your project setup
David, the serverless.yaml file (with long exclusion list reduced for sanity’s sake) is listed below. I will also add you to the repo too (it’s on BitBucket though … what’s your username). Here too are the files sizes of the various functions when I do a full deploy:
Note the stateInfo2 line item is a webpack bundled version of stateInfo but it doesn’t export the handler function correctly so doesn’t actually work (this I believe is down to some webpack/bundler issues that others are experiencing with some google-cloud api’s).
service: representative-truth-services
package:
individually: true
excludeDevDependencies: true
browser: false
exclude:
- node_modules/async-shelljs/**
- node_modules/chai/**
- node_modules/chance/**
- node_modules/coveralls/**
- node_modules/faker/**
- node_modules/inquirer/**
- node_modules/mocha/**
- node_modules/nyc/**
- node_modules/poi-preset-typescript/**
- node_modules/rollup/**
- (MORE)
custom:
stage: "${opt:stage, self:provider.stage}"
region: "${opt:region, self:provider.region}"
provider:
name: aws
runtime: nodejs6.10
profile: rt
stage: dev
region: us-east-1
functions:
stateInfo:
package:
exclude:
- node_modules/natural/**
- node_modules/sentiment/**
- node_modules/xml2js/**
environment: ${file(env.yml):${self:custom.stage}}
description: Get meta information on US States
handler: lib/endpoints-state/state-info.handler
timeout: 5
memorySize: 1536
stateInfo2:
package:
exclude:
- node_modules/**
environment: ${file(env.yml):${self:custom.stage}}
description: Get meta information on US States
handler: lib/endpoints-state/state-info-bundle.handler
timeout: 5
memorySize: 1536
federalBillsNew:
package:
exclude:
- node_modules/natural/**
- node_modules/sentiment/**
- node_modules/xml2js/**
- node_modules/cheerio/**
- node_modules/entities/**
- node_modules/html2plaintext/**
environment: ${file(env.yml):${self:custom.stage}}
description: Get all the recent federal bills from ProPublica and identify which are new to Representative Truth
handler: lib/endpoints-federal/bills/new.handler
timeout: 5
memorySize: 1536
federalBillsRefresh:
package:
exclude:
- node_modules/natural/**
- node_modules/sentiment/**
- node_modules/xml2js/**
- node_modules/cheerio/**
- node_modules/entities/**
- node_modules/html2plaintext/**
environment: ${file(env.yml):${self:custom.stage}}
description: Refresh the meta data for all federal bills
handler: lib/endpoints-federal/bills/refresh.handler
timeout: 5
memorySize: 1536
notDone:
package:
exclude:
- node_modules/**
environment: ${file(env.yml):${self:custom.stage}}
description: This function is not yet implemented
handler: lib/not-done.handler
timeout: 5
memorySize: 1536
# plugins:
# - serverless-step-functions
# - serverlss-pseudo-parameters
I was surprised by how inefficient and big wordnet-db is. It is a dependency of natural which is a natural language processing module I plan to use in a few functions but none that are here yet. I had been excluding natural at the function level but not word-net. The other obvious thing to exclude is aws-sdk as that’s given for free in Lambda.
Once doing that my file sizes are now listed below. In short, more manageable but the problem still remains.
I ran sls package and the analyized one of the functions (they’re all roughly the same size and should have same deps). Red areas are obvious devDeps that are included. There are more than this which are a bit more hidden. I’m disappointed my little script for identifying the deps didn’t pick it up but this was with Serverless’s excludeDevDependencies flag turned on too.
@DavidWells, I’ve added you to the repo. Try running npm run deps (after a yarn to get packages into place) for an analysis of which dependencies should be excluded. This is what is allowing me to operate right now.
With my “deps” script producing 482 exclusions, my stateInfo function is now 22.86mb (aka, a workable size). Here is the structure of the package that remains (google deps are by far biggest dead-code contributor … I am going to send the PRs for each of their repos which include node_modules as part of the repo … bad juju):
Note that I did add a few more exclusions to my always-exclude.yaml file (an input to the deps script) than the master branch has in it. If you want to get exactly the same result update this file to:
By comparison with no exclusions but using serverless’s package > excludeDevDependencies set to true the file size jumps to 74.62mb. I wanted to provide a visual map view but every time I try to SLS errors out with:
wordnet-db is 45MB and @firebase is 8.4MB. Get rid of those deps or tree shake out what you need and your zip should be much smaller. (this is 53.4 of the total 58.1mb zip)
I also removed the aws-sdk as it’s already included in the lambda runtime.
I will email you the updated serverless.yml and package.json I’m using.
