I am invoking a third party streaming API from Lambda which is giving me results in every 2 secs however lambda function has a execution limit of 5 mins which is causing my streaming API to stop. Is there a way to execute lambda function more then 5 mins or keep calling the streaming API after functionn stops ?
5 minutes is a hard limit from AWS. You can’t increase it.
is there any workaround to call the streaming API from Lambda function ?
I’m not sure exactly what you’re doing but it sounds a lot like “open a socket and keep writing to it” in which case the answer is no. You can have one Lambda invoke another to continue the process but the second Lambda would need to open a new socket.
1 Like
I would look into kinesis. AWS offers this as a streaming option so you could potentially invoke the kinesis service via lambda.
1 Like
Are you waiting for longer than 5 minutes for some sort of response, or are you trying to loop in the Lambda which is ending up taking too long ?
I had a scenario where I was using an API to create an object which took longer than 5 minutes, so I never saw the response back from that endpoint.
My workaround for that use case was to set a status in a DynamoDB row saying the creation had started, then have a cron Lambda execute every minute looking for those items that were creating, checking for the object, and when found updating the status in DynamoDB to be created.
Actually it is a streaming API which i need to run 24 hours to get the data, and it gives me data at interval of 2 secs. So, it only gives me data for 5 mins till function executes. I also tried scheduling lambda function every 2 minutes, but sometime it’s not working.
Nice Idea, can we call third party streaming API with Kinesis and then call Lambda function for operation on data ?
Most of the AWS apps are pretty proficient at talking to one another. You may need to setup some simple SNS (simple notification services) on AWS to get this all working.
If you’ve got a developer account on AWS I would also just ping one of their support engineers to get their two cents on setting this all up.
5 minutes is the current limit.
Shameless plug: I developed Containerless to enable workloads that don’t quite fit inside Lambda, but with (nearly) the same ease-of-use as a normal serverless function. Uses docker and ECS, and you can have long running tasks - we use it for this purpose.
Link: containerless.cool
2 Likes