This is probably a more general architecture question based around serverless tech, I have a bit of a vision in my head and I’m not sure if it’s a stupid idea, if I’m over-complicating things, or genuinely onto a real problem someone’s already solved. So please bare with me!
Fargate is a huge game changer as well as Lambda. In my current position we’re dealing with a lot of machine learning and big data tasks. For the front-end we’re using Lambda to power those UI interactions and dealing with metadata etc. But bigger operations we’re using Spark and Redshift etc. So we’re bridging the gap between our datascientists machine learning code, with our wider platform using containers, our case in Fargate.
So users are interacting with the ecosystem via Lambda, which will be making calls to Fargate containers to perform larger tasks which are too… hefty for Lambda.
I guess my question is, are there any direct integrations with serverless and Fargate that anyone knows of? Are there plans to support this (if anyone from Serverless Inc is reading this 
) or does anyone have any examples, or architectural diagrams of this working some place.
On the surface it’s pretty simple, right? You call a lambda, that has an ARN or Task Definition it calls. You could even use SNS, SQS or Kinesis to communicate between those two layers. However, I haven’t found a nice way to bridge the gap. I’m just using the AWS SDK to make calls and setting ARN’s as environment variables, stuff like that, which doesn’t feel too clean.
In a ‘traditional’ microservice stack, you have things like service discovery, or service meshes. Ultimately, I’d love to be able to use service discovery to call a service, and the client doesn’t know or care if that’s a Lambda or a Fargate container or not. But I’m not sure if that’s actually valuable or not, ultimately. I guess I’m just after a unified approach to using both Lambda and something like Fargate within AWS. Any suggestions or ideas greatly welcomed.