Single, small function to big to deploy?

yankeeinlondon · December 27, 2017, 5:31pm

I have a single Lambda function which is very small and has few npm dependencies and yet when I try and deploy I get the following:

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (85.9 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - representative-truth-services-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleStateMachineExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - StateInfoLogGroup
CloudFormation - UPDATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleStateMachineExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - StateInfoLogGroup
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - StateInfoLogGroup
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleStateMachineExecution
CloudFormation - UPDATE_COMPLETE - AWS::S3::Bucket - ServerlessDeploymentBucket
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - StateInfoLambdaFunction
CloudFormation - CREATE_FAILED - AWS::Lambda::Function - StateInfoLambdaFunction
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - representative-truth-services-dev
CloudFormation - UPDATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket
CloudFormation - UPDATE_COMPLETE - AWS::S3::Bucket - ServerlessDeploymentBucket
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - representative-truth-services-dev
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - StateInfoLambdaFunction
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleStateMachineExecution
CloudFormation - DELETE_COMPLETE - AWS::IAM::Role - IamRoleStateMachineExecution
CloudFormation - DELETE_IN_PROGRESS - AWS::Logs::LogGroup - StateInfoLogGroup
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - DELETE_COMPLETE - AWS::Logs::LogGroup - StateInfoLogGroup
CloudFormation - DELETE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - UPDATE_ROLLBACK_COMPLETE - AWS::CloudFormation::Stack - representative-truth-services-dev
Serverless: Operation failed!

The serverless error from this is:

An error occurred: StateInfoLambdaFunction - Unzipped size must be smaller than 262144000 bytes.

How can this be? For reference the function I’m uploading is:

import {
  IDictionary,
  AWSGatewayCallback,
  IAWSGatewayRequest,
  IAWSGatewayResponse
} from "common-types";
import GetStateInfo from "./vote-smart/get-state-info";
import { Lambda } from "aws-sdk";
import { STATES } from "./models/shared";
import DB from "abstracted-admin";
import { StateInfo } from "./models/StateInfo";
import { IApiResponse } from "./shared/ApiRetriever";
let _db: DB;
export const db = (setter?: DB) => {
  if (setter) {
    _db = setter;
  } else {
    if (!_db) {
      _db = new DB();
    }
    return _db;
  }
};

/**
 * Cycles through all states, asynchronously firing off a lambda function for
 * each. These worker functions will in turn update the Firebase DB for the given
 * state they are responsible for.
 *
 * @param event Event parameters passed down from AWS Gateway
 * @param context AWS params giving context to the execution
 * @param cb A callback function to call on exit or error
 */
export const handler = (
  event: IDictionary,
  context: IAWSGatewayRequest,
  cb: AWSGatewayCallback
) => {
  const lambda = new Lambda({ region: "us-east" });
  const states: string[] = event.states || [];

  if (states.length === 0) {
    cb(403, { statusCode: 403, error: "no states found in event" });
    return;
  }

  (async () => {
    const promises: Array<Promise<any>> = [];
    const errors: any[] = [];
    states.forEach((state: keyof typeof STATES) => {
      promises.push(processState(state, errors));
    });
    const result = await Promise.all(promises);

    cb(null, {
      statusCode: 200,
      body: JSON.stringify({
        messsage: `State Info retrieved for ${
          states.length
        } states: ${states.join(", ")}`,
        processed: states.length - errors.length,
        failed: errors.length,
        errors
      })
    });
  })();
};

async function processState(state: keyof typeof STATES, errors: any[]) {
  const api = new GetStateInfo().setState(state);
  const result: any = await api
    .getMapped()
    .catch(err => errors.push({ state, err }));

  return typeof result === "object"
    ? db().set<StateInfo>(`/reference/states/${state}`, result)
    : Promise.resolve();
}

it would be impractical for me include all the little files this refers to be please believe me it is rather minor. That said, I have not yet switched to a per function dependency configuration (I know you can do this but couldn’t find it yet in the docs). As a result there are a few more npm deps that are being brought in as you can see in the package.json:

{
  "name": "representative-truth-services",
  "version": "0.0.0",
  "description": "representative-truth-services",
  "license": "private",
  "repository": "",
  "author": "",
  "keywords": [
    "serverless",
    "typescript"
  ],
  "files": [
    "lib"
  ],
  "main": "lib/index.js",
  "typings": "lib/index.d.ts",
  "scripts": {
    "clean": "rimraf lib",
    "lint": "tslint --force --format verbose \"src/**/*.ts\"",
    "build": "ts-node ./scripts/build.ts",
    "run": "ts-node ./scripts/run.ts | ts-node ",
    "test": "ts-node ./scripts/test.ts ",
    "coverage": "nyc --reporter=text --reporter=html mocha --compilers ts:ts-node/register",
    "watch": "ts-node ./scripts/watch.ts",
    "deploy": "ts-node ./scripts/deploy.ts",
    "watch:test": "ts-node ./scripts/watch.ts test",
    "watch:all": "ts-node ./scripts/watch.ts all",
    "cli": "ts-node cli/index.ts --"
  },
  "dependencies": {
    "abstracted-admin": "^0.2.10",
    "aws-sdk": "^2.162.0",
    "axios": "^0.17.1",
    "cheerio": "^1.0.0-rc.2",
    "common-types": "~1.3.10",
    "entities": "^1.1.1",
    "firemodel": "^0.1.5",
    "html2plaintext": "^1.1.1",
    "moment": "^2.0.0",
    "natural": "^0.5.4",
    "sentiment": "^4.1.0",
    "xml2js": "^0.4.19"
  },
  "devDependencies": {
    "@types/chai": "^3.0.0",
    "@types/chance": "^0.7.33",
    "@types/cheerio": "^0.22.6",
    "@types/entities": "^1.1.0",
    "@types/faker": "^4.1.0",
    "@types/js-yaml": "^3.9.0",
    "@types/lodash": "^4.0.0",
    "@types/mocha": "^2.2.41",
    "@types/natural": "^0.2.33",
    "@types/node": "^6.0.0",
    "@types/rimraf": "^0.0.28",
    "@types/shelljs": "^0.7.1",
    "@types/stack-trace": "^0.0.28",
    "@types/xml2js": "^0.4.2",
    "chai": "^4.0.0",
    "chalk": "^2.3.0",
    "chance": "^1.0.0",
    "coveralls": "^2.0.0",
    "faker": "^4.1.0",
    "js-yaml": "^3.0.0",
    "lodash": "^4.17.4",
    "mocha": "^3.4.0",
    "nyc": "^11.0.0",
    "prettier": "^1.5.2",
    "rimraf": "^2.0.0",
    "serverless": "^1.24.1",
    "serverless-step-functions": "^1.3.0",
    "shelljs": "^0.7.0",
    "test-console": "^1.0.0",
    "ts-node": "^3.3.0",
    "tslint": "^5.0.0",
    "tslint-config-prettier": "^1.1.0",
    "typescript": "^2.7.0-dev.20171203"
  },
  "engines": {
    "node": ">=6.0.0"
  }
}

buggy · December 28, 2017, 1:16am

Inside the .serverless folder you should see the zip file that is being uploaded. Unzip it and check exactly what is being uploaded. You may find additional files and folders you’re not expecting that can be excluded.

If you can’t see anything obvious then check the size of your node_modules folder. It may be that one of your dependencies is actually very large or that adding up the size of all your dependencies and their dependencies is causing the problem.

Next, if you can survive with aws-sdk 2.143.0 then move it from dependencies to devDependencies. This will excluded it (and it’s dependencies) from the upload and you will use the version included with Lambda.

Finally, you may need to look at webpack and tree shaking to reduce the size of your files.

yankeeinlondon · December 28, 2017, 11:49am

Thanks @buggy, great suggestions. I’ll give that a try today. Especially like the aws-sdk one as I definitely do not need the latest. Do you happen to know where one can find information on configuring your npm deps on a function-by-function basis? This would be very helpful for me as each function will have a pretty different dependency graph.

yankeeinlondon · December 28, 2017, 12:00pm

Ok, I can see that the node_modules folder DOES indeed have way more than it should. It “should” only include my “dependencies” but it appears to also be including my “devDependencies”? For instance, things like Babel, etc. are being included. How does one avoid this?

buggy · December 28, 2017, 9:28pm

Can I assume you’re using an older version of Serverless? If you are then upgrade the version as they added support excluding devDependencies a while ago.

yankeeinlondon · January 2, 2018, 10:15am

No that’s just it. I remember when this wasn’t included but I have the latest version (1.25.0). I even – out of desperation – explicitly added:

excludeDevDependencies: true

to my serverless.yml file.

yankeeinlondon · January 2, 2018, 10:18am

In fact when I deploy it writes to the console that it is excluding devDependencies.

buggy · January 2, 2018, 10:59am

I’m starting to run out of ideas (short of a bug). Have you double checked your packages.json to make sure you haven’t accidentally added some as dependencies instead of dev dependencies?

Jari · January 3, 2018, 10:43am

This is how I do it to reduce the lambda size…

do not install node modules globally, not even serverless. Use npx to run serverless commands from local node_modules.

e.g npx sls deploy -v
In serverless.yml

Global settings for all lambdas

package:
individually: true
exclude:
    - '**/*'
include:
    - src/lambda/common_functions_for_all_lambdas/**

Single lambda

`package:
        include:
            - src/lambda/lambda_xyz/**

remember to run npm install for each lambda folder. I am using https://www.npmjs.com/package/serverless-plugin-scripts and separate npminstaller.js with package.json postinstall script to do that.

scripts:
hooks:
'before:package:createDeploymentArtifacts': npm install

Hope this helps!

(HOX! Check indentation, no time to play with this editor)

Lambda_Management_Console

chris · May 13, 2018, 8:35am

I found this plugin to be helpful for this: https://www.npmjs.com/package/serverless-plugin-include-dependencies
It cut my single file function (a very simple custom authorizer) down from 110MB to 52MB. However, that was still huge, and I realized it had the code from other functions in it (even though I have “package.individually” set to true. However, I think that may be because I have a mixed setup where my other functions use the Java runtime. So, I was able to add individual exclude rules to my JavaScript function to exclude the directories of the others. It’s via the normal package.exclude config, which you find more info on here: https://serverless.com/framework/docs/providers/aws/guide/packaging/. I wound up excluding a few more things and the final was down to only 180kb.

yankeeinlondon · May 17, 2018, 5:17pm

In the end I created a script that does static analysis on my dependencies and creates a list of safe directories to exclude. You can then use sls package to dig into the produced output and add or subtract a bit more. I’ve gotten my 50+ MB functions to consistently 4.2mb and I suspect if I spent a bit more time I could probably slim down a bit more (though I’ve already hit my 80/20 rule so not sure i’ll do this in the short term).

Topic		Replies	Views
Serverless tutorial Serverless Framework aws	3	1300	August 16, 2017
Sls deploy step function - how to upload each function individually Serverless Framework aws	1	848	January 7, 2019
How do you cleanly package small, individual Lambda Functions Serverless Framework	1	1754	June 26, 2017
Code structuring on large API / generic serverless AWS beginner questions Serverless Framework aws	2	1701	January 18, 2018
Serverless deploys full source code of the service for each lambda function Serverless Framework	3	1705	June 25, 2020

Single, small function to big to deploy?

Related Topics