Why does my brand new service get 1900 requests per hour consistently even throughout the night?


I was following the user-authentication tutorial for the serverless framework. I stopped after making the “createUser” function.

This is the only function I have, and I deployed it with just the POST method.

I have called this function maybe 5 or 6 times total since deploying it 48 hours ago.

The dashboard shows over GET 85,000 requests over the last 48 hours (consistently around 1900 requests per hour).

Where are these requests come from? They are all GET requests to the / route that 403 missing authentication, are these something from AWS? Does the serverless framework somehow have a retry policy on some healthcheck or something that constantly runs? Is there some test failing and thus it keeps retrying? I find it hard to believe that these are organic requests actually from legit sources by how uniformly they are spaced out.

Lastly, are these going to start costing me money because they are hitting the API Gateway where i’m paying for requests?

That doesn’t sound right, there is no retry policy in the serverless framework that I know of.

If it is still doing this then I would recommend running sls remove to decomission your AWS APIs etc as you will eventually get charged for this.

Can you post a link to the tutorial you followed?

It is possible it is just an automated tool run by anyone with the time to download it and do some basic config against your endpoint. It doesn’t have to be intentional, a lot of these tools can just iterate through possible domain values and make requests.

I would recommend getting in touch with AWS to investigate further what the cause may be as well. The Serverless Frameworkmakes no additional requests your endpoints at all; all metrics needed are retrieved via a CloudWatch subscription that happens asynchronously in the background and in no way involves HTTP at all.

Ok, i tried to run sls remove, I had a custom domain mapped to the api, so the function failed and left my service in a broken state.

I went and removed the domain mapping from api gateway, and then tried to run sls remove again, but it already removed the s3 bucket, so my dashboard still showed the service, the lambda functions were removed, but the api was not removed and the dynamodb wasn’t removed either.
I manually deleted the api and the dynamodb table from the aws console, and tried to see if i could deploy again.

it started giving me an error Application not found, so i had to open the dashboard, delete and recreate the app to try a deploy again.

Now the deploy failed with the following:

Warned - no cfnRole set
details: http://slss.io/sg-require-cfn-role
Require the cfnRole option, which specifies a particular role for CloudFormation to assume while deploying.

I don’t remember having to add anything like that when going through the tutorial, so i’m wondering if i should nuke my repo and start over.

The specific message about cfnRole is just a warning based on one of the default safeguards set on your deployment profile. You can actually ignore that. Any issue with your deployment would be related to an error after that message

1 Like

I thought so, the actual error ended up being that the s3 bucket didn’t exist anymore and it was recommended that i manually create it. So i did that and did another deploy, which seemed to be successful at least to the point that the output said it was pushed to the dashboard, but i don’t see the api in api gateway. It also seems to have the exact same url slug as the old one that i removed, i’m wondering if there is a way to force it to completely generate a new service.

The tutorial is here: https://www.youtube.com/watch?v=VlyaZuY_XLk (this video and the one with #17 User Creation in it’s title that is part of the same playlist)

Now i’m getting:

Stack:arn:aws:cloudformation:us-west-2:222008606357:stack/user-service-dev/f5563230-c176-11ea-82bd-0a5d164d0af8 is in DELETE_FAILED state and can not be updated.

even using sls deploy --force doesn’t seem to help.

I renamed the app and service to something else in my serverless.yml and the deploy functions perfectly.
So something must have gotten out of sync.

If you go to the CloudFormation service inside AWS, you can manually remove the service. You may need to jump through a few hoops with deleting of specific resources in the CloudFormation stack, but once that is deleted, a new deploy from that point on is a whole new stack.

You could also just change the service name slightly to force creation of a new stack.