Using aws lambda to concatenate small files in S3 into multi GB file

Serverless Framework
1

Hi, I want to merge multiple small files into multi GB file using lambda, I understand lambda has certain limits on memory sizes and run time. Can this be done with Lambda? or what is the Best way to do this?
Thanks in advance!

2

This domain-specific bioinformatics paper might give you an idea on how to proceed:

Just look at the (simple) method:

  1. Pull from S3 to EC2.
  2. Split files.
  3. Upload splitted files to S3.
  4. Run lambdas on the splitted files.
  5. Pull results from the lambdas to EC2 for merging.

Hope that helps!