Possible Bug With serverless invoke

lambda
#1

I may have found an issue with “serverless invoke”.

I have a long running lambda (lambda-a) which itself invokes another lambda (lambda-b) multiple times, then returns the results.

The problem is using “serverless invoke” when trying to test my deployed lambda by invoking it from my local system. For example…

serverless invoke -l t -s dev -f lambda-a -p mydata.yml

I have my function timeout set to 15 minutes.

When the work lambda-a is doing completes in under 2 minutes, things seem to be just fine. When it takes over two minutes, something strange happens. At about the 2 minute mark, my lambda-a function gets called a second time (based on logging I am doing.) It creates a new request ID, which shows in the logs. So, my logs look something like this…

START RequestId: a96ec085-4eb5-4a2e-be08-5e5c6637ba0b Version: $LATEST
2019-05-14 15:15:41.898 (-07:00)	a96ec085-4eb5-4a2e-be08-5e5c6637ba0b	My log statement A
2019-05-14 15:15:41.898 (-07:00)	a96ec085-4eb5-4a2e-be08-5e5c6637ba0b	My log statement B
... (about 2 minutes later)
START RequestId: 07a550f6-5870-4b45-8a32-469917389c7f Version: $LATEST
2019-05-14 15:17:42.223 (-07:00)	07a550f6-5870-4b45-8a32-469917389c7f	My log statement A
2019-05-14 15:17:43.387 (-07:00)	a96ec085-4eb5-4a2e-be08-5e5c6637ba0b	My log statement C
2019-05-14 15:17:43.898 (-07:00)	07a550f6-5870-4b45-8a32-469917389c7f	My log statement B
2019-05-14 15:17:44.100 (-07:00)	07a550f6-5870-4b45-8a32-469917389c7f	My log statement C
END RequestId: 07a550f6-5870-4b45-8a32-469917389c7f
REPORT RequestId: 07a550f6-5870-4b45-8a32-469917389c7f	Duration: 121499.07 ms	Billed Duration: 121500 ms 	Memory Size: 3008 MB	Max Memory Used: 99 MB

Notice the function starts a second time at about 2 minutes. Then the logs are mixed.

When I invoke this from AWS console, this does NOT happen. It runs and completes successfully, and does not invoke the function a second time at the 2 minute mark.

I have also gotten some intermittent warnings from serverless a few times about a “recoverable error - 120000 ms” (but not always). I am not sure of the exact message as I have not gotten it recently.

I feel like this might be something specific with how “serverless invoke” works since I do not get this problem when invoking from AWS console. Also, the warning message that sometimes appears seems odd.

Really, I want to see if others have had this experience or if someone with more in-depth knowledge of the “serverless invoke” command knows anything.

Thanks!

#2

I figured this out. It was an http timeout occurring from within the AWS client. The solution is to set AWS_CLIENT_TIMEOUT environment variable.

AWS_CLIENT_TIMEOUT=360000 serverless invoke -l t -s dev -f lambda-a -p mydata.yml

This sets the httpOptions.timeout configuration for the AWS SDK.