I may have found an issue with “serverless invoke”.
I have a long running lambda (lambda-a) which itself invokes another lambda (lambda-b) multiple times, then returns the results.
The problem is using “serverless invoke” when trying to test my deployed lambda by invoking it from my local system. For example…
serverless invoke -l t -s dev -f lambda-a -p mydata.yml
I have my function timeout set to 15 minutes.
When the work lambda-a is doing completes in under 2 minutes, things seem to be just fine. When it takes over two minutes, something strange happens. At about the 2 minute mark, my lambda-a function gets called a second time (based on logging I am doing.) It creates a new request ID, which shows in the logs. So, my logs look something like this…
START RequestId: a96ec085-4eb5-4a2e-be08-5e5c6637ba0b Version: $LATEST
2019-05-14 15:15:41.898 (-07:00) a96ec085-4eb5-4a2e-be08-5e5c6637ba0b My log statement A
2019-05-14 15:15:41.898 (-07:00) a96ec085-4eb5-4a2e-be08-5e5c6637ba0b My log statement B
... (about 2 minutes later)
START RequestId: 07a550f6-5870-4b45-8a32-469917389c7f Version: $LATEST
2019-05-14 15:17:42.223 (-07:00) 07a550f6-5870-4b45-8a32-469917389c7f My log statement A
2019-05-14 15:17:43.387 (-07:00) a96ec085-4eb5-4a2e-be08-5e5c6637ba0b My log statement C
2019-05-14 15:17:43.898 (-07:00) 07a550f6-5870-4b45-8a32-469917389c7f My log statement B
2019-05-14 15:17:44.100 (-07:00) 07a550f6-5870-4b45-8a32-469917389c7f My log statement C
END RequestId: 07a550f6-5870-4b45-8a32-469917389c7f
REPORT RequestId: 07a550f6-5870-4b45-8a32-469917389c7f Duration: 121499.07 ms Billed Duration: 121500 ms Memory Size: 3008 MB Max Memory Used: 99 MB
Notice the function starts a second time at about 2 minutes. Then the logs are mixed.
When I invoke this from AWS console, this does NOT happen. It runs and completes successfully, and does not invoke the function a second time at the 2 minute mark.
I have also gotten some intermittent warnings from serverless a few times about a “recoverable error - 120000 ms” (but not always). I am not sure of the exact message as I have not gotten it recently.
I feel like this might be something specific with how “serverless invoke” works since I do not get this problem when invoking from AWS console. Also, the warning message that sometimes appears seems odd.
Really, I want to see if others have had this experience or if someone with more in-depth knowledge of the “serverless invoke” command knows anything.
Thanks!