Heap out of memory

Serverless Framework
1

Hi there,

I am using xlsx (https://www.npmjs.com/package/xlsx) in order to parse a large .xls file (about 1Mb).
I was able to do that previously in a lambda, and I’m trying to migrate everything to serverless.
I am using serverless-offline with serverless-webpack to transpile from ES6 with async await to node 6.
However, it seems that I have a “Javascript heap out of memory”…

I also tried to increase the memory by running like this:
node --max-old-space-size=12000 ./node_modules/serverless/bin/serverless webpack invoke -f import -p src/functions/import/event.json

It does not change anything. Did anyone already encountered the same issue? What can I do?

Thanks

Here is the stacktrace:

<--- Last few GCs --->

   34509 ms: Mark-sweep 1341.7 (1395.0) -> 986.9 (1040.1) MB, 319.9 / 0 ms [allocation failure] [GC in old space requested].
   35147 ms: Mark-sweep 986.9 (1040.1) -> 986.9 (1040.1) MB, 638.4 / 0 ms [allocation failure] [GC in old space requested].
   35638 ms: Mark-sweep 986.9 (1040.1) -> 986.5 (1034.1) MB, 491.2 / 0 ms [last resort gc].
   35934 ms: Mark-sweep 986.5 (1034.1) -> 986.5 (1033.1) MB, 295.2 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1a600d2c9e31 <JS Object>
    2: make_sector_list [/Users/myuser/path/to/my/pkg/node_modules/xlsx/xlsx.js:~1259] [pc=0x32a8c9e4023c] (this=0x1a600d2e15d9 <JS Global Object>,sectors=0x330c67670d29 <JS Array[417]>,dir_start=414,fat_addrs=0x330c67670d09 <JS Array[4]>,ssz=512)
    3: parse [/Users/myuser/path/to/my/pkg/node_modules/xlsx/xlsx.js:1090] [pc=0x32...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/myuser/.nvm/versions/node/v6.3.1/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/myuser/.nvm/versions/node/v6.3.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/myuser/.nvm/versions/node/v6.3.1/bin/node]
 4: v8::internal::Factory::NewUninitializedFixedArray(int) [/Users/myuser/.nvm/versions/node/v6.3.1/bin/node]
 5: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::ConvertElementsWithCapacity(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::FixedArrayBase>, v8::internal::ElementsKind, unsigned int, unsigned int, unsigned int, int)     [/Usmyuseruser/.nvm/versions/node/v6.3.1/bin/node]
     6: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::GrowCapacityAndConvertImpl(v8::internal::Handle<v8::internal::JSObject>, unsigned int) myusers/myuser/.nvm/versions/node/v6.3.1/bin/node]
     7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/myuser/.nvm/versions/node/v6.3.1/bin/node]
     8: 0x32a8c8806338
    [1]    9066 abort      serverless webpack invoke -f import -p src/functions/import/event.json
2

Note that sometimes the error is totally different (looks like a xlsx error, but I’m almost sure it’s still a memory issue, never had before). And also, sometimes, execution goes fine, and I have the expected result.

Here is the second error I can have

Error in handler: TypeError: Cannot set property 'name' of undefined
    at parse (/Users/path/to/my/pkg/node_modules/xlsx/xlsx.js:1092:29)
    at Object.readSync [as read] (/Users/path/to/my/pkg/node_modules/xlsx/xlsx.js:1354:9)
    at readSync (/Users/path/to/my/pkg/node_modules/xlsx/xlsx.js:17412:34)
    at Object.readFileSync (/Users/path/to/my/pkg/node_modules/xlsx/xlsx.js:17430:9)
    at parseReport (/Users/path/to/my/pkg/.webpack/src/functions/import/index.js:816:23)
    at _callee$ (/Users/path/to/my/pkg/.webpack/src/functions/import/index.js:177:28)
    at tryCatch (/Users/path/to/my/pkg/node_modules/regenerator-runtime/runtime.js:65:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (/Users/path/to/my/pkg/node_modules/regenerator-runtime/runtime.js:303:22)
    at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/Users/path/to/my/pkg/node_modules/regenerator-runtime/runtime.js:117:21)
    at step (/Users/path/to/my/pkg/node_modules/babel-runtime/helpers/asyncToGenerator.js:17:30)
    at /Users/path/to/my/pkg/node_modules/babel-runtime/helpers/asyncToGenerator.js:28:13
    at run (/Users/path/to/my/pkg/node_modules/core-js/library/modules/es6.promise.js:87:22)
    at /Users/path/to/my/pkg/node_modules/core-js/library/modules/es6.promise.js:100:28
    at flush (/Users/path/to/my/pkg/node_modules/core-js/library/modules/_microtask.js:18:9)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

  Type Error ---------------------------------------------

  Cannot set property 'name' of undefined

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.3.1
     Serverless Version:     1.18.0
1 Like
3

Never mind, I thought it was linked to serverless because it didn’t happen before, but I also upgraded my node version.
I found the issue (was retrieving a xlsx file with http.get via a pipe, and was trying to parse it on request.on('end') event instead of pipe.on('finished') so sometimes the file was not totally closed when I opened it…)