Hitting CloudFormation Max Resources


#1

I understand that using CF makes things easier for the severless framework. However, this also means serverless’ limits are directly tied to the limitations of CF. Since serverless makes creating APIs easier, developers are going to want to create complicated APIs with serverless. This means developers will most likely hit the limits of CF; specifically the max 200 resources limit.

Since the developer is creating an API, using multiple serverless projects isn’t going to work (if it would work then please let me know how). The only potential solution I can see is the use of nested CF stacks. It doesn’t appear that serverless supports this out-of the-box. I need to use the Resources section of the serverless file. Would it be possible to provide some documentation on the best practices to accomplish this task? I think it is worth some documentation since it seems to me many users of serverless will hit this problem if they are developing any meaningful API.


#2

The resources section in serverless.yml is just CloudFormation. All the CFN docs apply to it.

Another way to handle this (that avoids the need for nested stacks) is cross-stack references. I avoid nested stacks like the plague.

What resources are you creating so many of that you’re hitting the limit? Individual functions?
Do you have a 1:1 mapping between endpoints/methods and functions?

I use CloudFormation every day for all my projects (Serverless and otherwise) and still haven’t hit this limit, so we must be doing something very different.


#3

yes, i get the feeling I am doing something wrong. For my api there is a one to one mapping between api endpoint/method and lambda function. But that is all it is. I assume that each “get”, “put”, “delete” etc. is a resource. right now I have roughly 46 methods on roughly 15 endpoints. each method maps to a lambda function. However, even at 46 methods, why are there over 200 resources?


#4

Serverless has to create a bunch of “background” resources to glue the other resources (e.g. functions, methods, etc) together, so there’s definitely going to be more than you think.

The best way to see all the resources is go to the CloudFormation section of the AWS Console, select your stack and go to the “Resources” tab.

Here’s an AWS CLI command will print out an ordered summary of how many of each resource has been created:

aws cloudformation describe-stack-resources --stack-name [YourStackName] --query "StackResources[].ResourceType" --output text | tr "\t" "\n" | sort | uniq -c | sort -r

I’d be interested to see what you’ve got, if you can share.

Unfortunately I think you’ll still be left with few options: A) Split in to multiple services B) Re-use functions for multiple endpoints :frowning2:

Update: I realised the first version of the command I gave you didn’t work…


#5

I think @rowanu is correct, with 46 methods you have not structured your project correctly, it is dead easy to reference a resource that is output by one CloudFormation in another, serverless or otherwise.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-crossstackref.html

You need to break up your solutions into much smaller projects / services


#6

It is wrong to have a service with 46 methods? I don’t think I’ll go with that.

The idea that no service should be so complicated that it has 46 methods doesn’t seem realistic. I know I could break it up into smaller services, but it seems to me that breaking up a service simply because the provider (aws) has a limitation, and not because it makes some kind of logical sense, is not a great idea either.


#7

There’s a discussion about this going on at https://github.com/serverless/serverless/issues/3411