I’d be interested in hearing how you all are dealing with debugging. Lambda errors are notoriously difficult to interpret and contain minimal information. I’m using Atom which has little in the way of syntax checking. Debugging has been a challenge for me. How are you dealing with it?
I run my functions locally, and if I get syntax errors I enable
SLS_DEBUG=* to get more detailed information. If the functions are running, but I’m getting service failures, I’ll dump errors to the logs in debug mode.
What types of errors are you trying to debug? Connecting to AWS services? Syntax errors? Are you running a test suite to unit test your functions? What language are you using?
what type of errors you are encounting…! I need to add one more question to this… I.e., Run-time Errors.
Actually, “A run-time error is generated when a VBA program attempts to execute an invalid statement at run time.”
If you are clear about what type of errors you are getting. It will be easy to get your answer…
I don’t have a clear answer, but I’ll share my experience so far with debugging functions.
I’m currently building a medium sized platform with serverless, using typescript + webpack.
I opted to do integration tests instead of unit tests in order to have some sort of flows covered. For this I piggybacked on webpack and I use the bundles it creates and run them locally in some plain node.js scripts.
This way I can build flows using responses from one function to another. I’m using VSCode and its debugger is very helpful when it comes to environment variables.
Obviously there are some catches to this approach:
Catch 1 - My tests are not covering the serverless.yml configuration
Since I’m using only the code that will be deployed to the Lambda function, I often encounter errors generated by serverless.yml misconfigurations, mostly forgotten environment variables.
Possible solution: I created flows in postman that work directly with API that would reveal any endpoint misconfiguration and I can use them as working examples and for demo purposes.
Catch 2 - A webpack generated bundle is a bit hard to debug
Webpack bundles everything in one file. I tried to keep the number of modules used at minimum, even avoided using lodash , but still one file has 1-2 MB. You can imagine that these files are a bit hard to debug but not impossible.
Possible solution: I’m thinking to transpile the typescript code to js and use those files without bundling and tree shaking.
Catch 3 - The process of preparing the files for testing is long
The bundle creation time increases with every new function you add. I have now 60 functions that are bundled every time I need to run the tests, and it takes 4 minutes to finish. I have to run this after every update I do to the functions code, so TDD is not easily achievable.
Possible solution: Bundle only the functions needed for a certain test. This way I don’t have to bundle every thing every time. This way I can get a shorter bundling time and still use the final product that would run in the lambda function. The downside is to configure every test with what needs to bundle.
I know this is not probably the answer you were looking for, but it might give you an insight on how others are doing it. Also I think this course might help with debugging and in general about using serveless https://livevideo.manning.com/course/38/production-ready-serverless?
@IulianGioada great answer. enlightening. I’ll share how I’m doing tests and the problems I encounter.
A bit about my stack:
- Python + Django
- Using Zappa
- My AWS services: S3, Lambda, RDS, ES
As you can see I have 2 services that are not completely serverless, they are managed, but there is an actual server behind the scene, it greatly affects my testing strategy.
I use the standard testing pyramid, e.g. unit testing my Lambda functions, integration tests for my end to end testing and exploratory testing to find hidden “gems”. Pay attention that except unit tests I run all of my tests in real environment, I try to avoid mocks and simplify my dev environment and components I’m dealing with. For actual debugging during exploratory testing I use CloudWatch --> ES integration to send my logs for improved digestion. CloudWatch is horrible and unusable, ELK is a god sent service.
The issues that I encountered during testing:
- Each of our developers have a separate environment to test their code in live cloud environment, it’s expensive, each one needs to provision RDS + ES, using AWS organisation features does not really help, in addition provision each time a new environment to a new developer is a nightmare.
- Network is a real problem, as you mentioned preparing, uploading the fixes take a lot of time, it’s really annoying and takes the fun out of it.
@IulianGioada If you’re using Typescript & webpack then I can recommending
- Use the webpack-node-externals plugin so that it doesn’t bundle your node_modules. I’ve had problems with this.
- Enable source maps
- Add the source-map-support NPM package as a dependency
import "source-map-support/register";to the top of your files
- Set a minimum of 256MB for your Lambda
- Don’t use tree shaking. It will make your files a lot smaller but it mucks with source maps.
I’m using Node.js Lambdas that need to take weird real-world raw input events from fairly dumb IoT devices and then do some processing before sending them on to a legacy system. It’s a bit hacky and local debug weren’t doing the job because they were so different from the real use-cases.
I switched over to using Rookout for live Lambdas a while back. It lets you do step debugging on the Lambda by setting oldskool breakpoints that dump the stack without stopping the function execution.
Adding my 2c here. I have found unit tests the easiest way to debug locally. To run your Lambda you will probably need some kind of wrapper script to pass it parameters to execute so you might as well make it a unit test. Now you can execute tests with breakpoints any time you want. There are a number of node modules I use to help with this:
- serverless-mocha-plugin: Helps wrap your lambda in a way that makes passing it event data easy as well as then performing assertions.
- aws-sdk-mock: You often need to communicate with AWS services but when running tests you don’t want to do this. Also, its difficult to test AWS service failure if you are pointing at a live service so this module lets you simulate specific responses
If you want anymore clarification feel free to ask