Layers upload on each deploy?

Serverless Framework
1

AWS Layers support was added in v1.34. How it seems to work right now is that every time you deploy your serverless app the layers are packaged and uploaded. However, this is not what I expected. I thought we would only want to upload/update the layer package if a file was added/removed or changed.

Is there a reason it is implemented this way? I think my deployment time would be cut down by a minute or two if we didn’t upload unchanged layers because some of my layers are large (100mb +) so they take a while to upload.

1 Like
2

I agree. Also, every time a deploy is made, a new version of the layer is generated even if we don’t make any change to it.

3

You’re right about that. Looking into it now.

For the time being, one way to handle this is to have your layer & your service using the layer in different serverless projects.

In your service’s serverless.yml:

service: foobar
provider:
  name: aws
  language: python3.6
functions:
  hello:
    handler: handler.hello
    layers:
      - ${cf:foobar-layer-dev.FoobarLayerExport}

Then in a separate project managing your layer:

service: foobar-layer
provider:
  name: aws
layers:
  foobar:
    path: layer
resources:
  Outputs:
    FoobarLayerExport:
        Value:
          Ref: FoobarLambdaLayer
        Export:
          Name: FoobarLambdaLayer
4

Update: I’ve tried tweaking the cloudformation template in subsequent uploads so that the layer definition is identical. But it still creates a new layer version. As such, the work around above is your best solution it seems.

5

+1, having layers deployed only when they change would be really nice. The ffmpeg example in the blog post is about 50MB, which is quite long to upload and layers could definitely be the solution to upload it only once.

6

+1, hope to keep this conversation alive. Redeploying layers every time seems to eliminate a huge advantage of using layers to begin with.

8

I’ve done this but I redeploy my layer (creating a new version) and my service still points to the previous version of the layer, how can I solve this? without having to redeploy my service?

9

+1 this is causing a huge issue blocking CI deployments. Is this resolved yet?

10

It has a plugin which solves this problem.

1 Like
11

I noticed this too. I think (IMO) it makes more sense to just not even use Serverless at all when creating layers. I’ll probably just manually create a layer to hold my audio binaries (ffmpeg, etc), and then use the ARN in my Serverless stacks to use those binaries. That way, if I recompile ffmpeg, I can create a new version, write in the description about what version of ffmpeg it is (or whatever), and then just update the Serverless config to point to the new version.

12

I was able to avoid the Excluding development dependencies by zipping the layers manually and referencing them with package>artifact>layer.zip , this helped me to reduce deployment time by 30 sec

Project structure:

.
├── babel.config.js
├── env
├── jest.config.js
├── layers    # <---
├── LICENSE
├── node_modules
├── package.json
├── README.md
├── scripts
├── serverless.yml
├── services
├── tests
└── yarn.lock

My layers folder:

.
├── axios
│   ├── axios.zip
│   └── nodejs
│       ├── node_modules
│       ├── package.json
│       └── yarn.lock
└── nodemailer
    ├── nodejs
    │   ├── node_modules
    │   ├── package.json
    │   └── yarn.lock
    └── nodemailer.zip

serverless.yml layers part:

layers:
  nodeMailer:
    package:
      artifact: layers/nodemailer/nodemailer.zip
  axios:
    package:
      artifact: layers/axios/axios.zip

The plugin that was mentioned earlier agutoli/serverless-layers didn’t work for me, I get a weird error of “could not push of undefined”

My current solution does not resolve the fact that the layers are being created+deleted each deployment.

I think I figured out why this is happening, after reading this - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-layerversion-content.html
The Lambda Layer Version is referencing an S3 object, this means that each time we deploy we upload a new S3 object, so each time we create a new lambda layer version.

I assume that this can be solved by creating the lambda layers manually and decouple them from the serverless.yml file :\

13

I agree that the solution is to have your layers in another project. One huge reason to use layers is to share code, like writing your DB connection pooling once, deploy it as a layer, and have all your other projects use the layer to get connections. It’s great for auth layers too so you don’t have to copy paste your auth code everywhere.

Keeping them in the same project is nice while you learn them, but to share them you’ll find you’ll eventually want to move them so you’re not always trying to figure out which app project the DB layer is in then having to redeploy that project to get DB changes out.

1 Like
14

agreed, lesson learned