Best way for offline workflows (step functions, sqs)

Serverless Framework
1

I am trying to convert my long running lambda functions into separate workflows using step functions and/or sqs. My main requirement is that we have to run these locally and verify the workflow before we deploy.

My workflow is:
API Gateway -> Lambda (step-function) -> Lambda (step-function) -> Lambda -> HTTP (external API)

Right now I have a working API Gateway -> Lambda -> HTTP, but there is so much processing going on that I have been running into the occasional 30 second timeout issues on the API Gatway’s end. Since the clients cannot support (at this time) Websockets, thats not an option right now.

Alternate workflow:
API Gateway -> Lambda -> SQS -> Lambda -> SQS -> Lambda -> HTTP

Option 1: the various offline step function plugins.
So far I spent most of the day working with the serverless-offline-step-functions plugins but its very buggy and has some limitations.

  1. I cant seem to trigger it from an existing API gateway. It must be triggered from a separate port.
  2. When I do trigger it. I get errors “undefined: 1” (I’ll report that to their own github site)
  3. no way to monitor the workflows that I can see.

I looked at a few other plugins:
https://github.com/pianomansam/serverless-offline-stepfunctions-local with https://github.com/pianomansam/serverless-offline-stepfunctions-local and
https://www.npmjs.com/package/serverless-step-functions-offline
but before I spend more time on those, I was wondering if there was one that does a great job before I test each one only to find they don’t work.

Option 2: localstack

I have not tried that but it seems like it could work once you get through all the configuration steps.

Option 3: offline sqs
There are a few plugins here too. But like the step-function plugins, I wonder how well these are maintained and what their limitations are. It seems most of these cannot be passed from lambda to sqs but rather must be triggered from a mock sqs command.


It seems the only way to get a real understanding of how the application will work is to simply use real deployed sandbox stacks in AWS, but we have some restrictions in place that make that difficult.

So for those who have gone before… what is the best way to approach this?

2

Be aware that you’ll still have 30 sec timeout on API Gateway -> Lambda as well as on API Gateway -> Lambda (step-function) (not 100% sure about this one)
So if you cant use websockets - it still wont be enough to get the result. If you just want to trigger the processing - it would work.

For testing, localstack should work for you.

3

Thanks. The main idea was to hand off execution and simply return a success with a step function, so rather than wait for a response that something succeeded, it just needs to hand off to another script. When its done it will return to a separate HTTP endpoint to log the success or failure, etc. So the HTTP at the end was a different endpoint not a return response.

We are looking into converting to websockets. Is it possible via AWS or Serverless to simply convert an existing API gateway from HTTP to websockets or do we have to rebuild a new gateway?

4

Cant answer this, unfortunatelly.
We’ve moved to AWS AppSync (GraphQL) with its subscriptions when we needed this feature.