I have to build architecture to curl all the html pages on my prelive environment and save the static content to S3.
I need to limit the number of parallel curl requests running against my prelive servers to avoid DDOSing my prelive environment.
As it might take a while (1,000’s of URLs) I want to provide the user feedback on the progress of the curl requests.
I had an idea to post the URLs to a Lambda master function calling Lambda slaves using the fanout method dividing out curl requests. Unfortunately this does not give the user feedback as the lambda task does not return with a response until it is finished.
Any idea how I might do this in the serverless environment or how that might fit within a serverless / hybrid environment?