My goal is to come up with an optimal LLM API vendor workflow for my serverless application.
Considering that my lambdas are my application layer, calling an LLM API from a lambda causes the lambda to sit idle, wasting time, waiting for the LLM response. Considering that LLMs can take minutes to respond, you can see how impactful this becomes.
I thought of delegating the LLM calls to Step Functions, but they have a payload limitation of 256KB, which is also a blocker for some of the responses.
Had the LLM vendors supported webhooks, it would’ve been ideal, but they unfortunately don’t.
So, how do we avoid waiting for the LLM API response?