I am struggling to see how to get parallelism for Lambda consuming from MSK topic. The scenario is that we have a Lambda event source to consume from MSK single topic.
What we have noticed that there is only a single Lambda instance is running even though the concurrency is set to a high number. Not sure this is a restriction or not. Basically, we have millions of messages to be processed and here is more like a sequential process and that is not good enough for us.
How to get multiple lambda instances consuming from same topic - like proper lambda parallel process.
Any thoughts would be highly appreciate. If this not please suggest an alternative - SQS message has a payload size limit as well