Node process runs out of memory

[thelgevold]

I have been trying to integrate the JavaScript version of the Closure compiler with my application as a Rollup plugin.
The process works for simple projects, but I have observed that the process runs out of memory in bigger projects:

The process runs for a while, but crashes eventually with the error message:

<--- Last few GCs --->

  245503 ms: Mark-sweep 13967.5 (14023.0) -> 13966.5 (14023.0) MB, 14259.4 / 0 ms [allocation failure] [GC in old space requested].
  262655 ms: Mark-sweep 13966.5 (14023.0) -> 13966.5 (14023.0) MB, 17154.0 / 0 ms [allocation failure] [GC in old space requested].
  280443 ms: Mark-sweep 13966.5 (14023.0) -> 13966.4 (14023.0) MB, 17789.0 / 0 ms [last resort gc].
  298983 ms: Mark-sweep 13966.4 (14023.0) -> 13966.4 (14023.0) MB, 18541.4 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x298b1a5b4629 <JS Object>
    1: qhb(aka qhb) [/Users/helgevot/temp/angular-2-samples/node_modules/google-closure-compiler-js/jscomp.js:~4620] [pc=0x1f9c9b11dbc4] (this=0x298b1a5041b9 <undefined>,a=0x178543a1ed51 <a uhb with map 0x1bcd61950b81>,b=0x71dd11e34f1 <a thb with map 0x1bcd61951209>)
    2: aec [/Users/helgevot/temp/angular-2-samples/node_modules/google-closure-compiler-js/jscomp.js:6280] [pc=0x1f9c9b110535] (t...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
sh: line 1: 19438 Abort trap: 6           npm run rollup

Here is my rollup code that integrates the "google-closure-compiler-js": "^20160828.0.1" package

var closure = require('google-closure-compiler-js');

function closureCompilerPlugin(){
  return {
    transformBundle(bundle){
      var transformed = closure.compile({jsCode: [{src: bundle}]});
      return transformed.compiledCode;
    }
  }
}

export default {
    entry: 'main.js',
    dest: 'dist/bundle.js',
    sourceMap: false,
    format: 'iife',
    plugins: [
        commonjs({
            include: 'node_modules/rxjs/**',
        }),
        nodeResolve({
            jsnext: true, module: true
        }),
        closureCompilerPlugin()
    ]

}

I think the Java version will work, but I would really like to get this working with the JS version since the setup is so much simpler.

[thelgevold]

The project I am targeting is an Ahead of Time (AoT) compiled Angular 2.0 application

[samthor]

Thanks for the bug report. We've tested the project on medium-sized builds but obviously did not go far enough.

Can you post repro instructions? Do you have a branch I can check out and run build?

[thelgevold]

@samthor Thanks for the quick response.
I have my closure-compiler branch here https://github.com/thelgevold/angular-2-samples/tree/closure-compiler

Here are the repro steps

1. npm install
2. npm run install-application

The second step will first run the Angular 2 Ahead-of-time compiler on the Angular source. Next the result of that is fed into rollup where the closure compiler will be triggered via the plugin.

My observation has been that the process runs for a few minutes before the error occurs.

[steveblue]

I can confirm this issue happens in even a modest hello world example of Angular 2 compiled AOT. process out of memory happens every time I try to implement JS version with multiple versions of node. I attempted to use "google-closure-compiler-js": "^20160828.0.1" with node 4.5.0,5.5.0, and 6.5.0 with OS X El Capitan 10.11.6. I could get the .jar to compile every time, but when trying to implement with Rollup like in the example above process out of memory happens after a few seconds after starting to compile.

[concavelenz]

Are you giving more than the default heap?

[thelgevold]

@concavelenz I am not 100% sure what you mean. I have tried setting max_old_space_size. Is that what you're referring to?

max_old_space_size didn't seem make a difference as I tried setting it to a high value, but still ran out of memory.

[concavelenz]

yes, what values did you try?

On Sun, Sep 18, 2016 at 10:51 AM, Torgeir Helgevold <
[email protected]> wrote:

@concavelenz https://github.com/concavelenz I am not 100% sure what you
mean. I have tried setting max_old_space_size. Is that what you're
referring to?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#23 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABMDKj155dG8IyLrFFkHddBbdd_TwFbyks5qrXo8gaJpZM4J-NVW
.

[thelgevold]

I tried a few different values. I remember setting to 8000 and 14000 on a Mac with 16GB of Ram

[danxinju]

@samthor Same problem appeared, it would be great if you have a solution to make the large-sized js works ok.

[Kurren123]

I'm getting the same problem.

<--- Last few GCs --->

[13028:0000025B0907E770]    25624 ms: Mark-sweep 1387.5 (1397.5) -> 1387.5 (1397.5) MB, 378.0 / 0.0 ms  allocation failure GC in old space requested
[13028:0000025B0907E770]    26003 ms: Mark-sweep 1387.5 (1397.5) -> 1387.5 (1396.5) MB, 378.9 / 0.0 ms  last resort
[13028:0000025B0907E770]    26383 ms: Mark-sweep 1387.5 (1396.5) -> 1387.5 (1396.5) MB, 379.8 / 0.0 ms  last resort


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000000D8146A66A1 <JS Object>
    1: Jjb [C:\Users\Kurren.Nischal\Documents\DevelopmentTests\Optimise\node_modules\google-closure-compiler-js\jscomp.js:~4866] [pc=00000231DB935588](this=000000D8146FECC9 <JS Global Object>,a=0
00000751F9A4779 <a Pjb with map 000000BA5D9580A9>,b=00000069C692B161 <an Ojb with map 000000C75854AD99>)
    2: Dnc [C:\Users\Kurren.Nischal\Documents\DevelopmentTests\Optimise\node_modules\google-clo...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[samthor]

Can you please try again with Node v8.3.0? We believe that this version lifted the maximum value supported in max_old_space_size.

[danxinju]

@samthor Thanks for your replay, but my real intention is that we have no much space for the closure compiler, and java version of closure compiler is ok.

So I simply think that the final solution is that javascript version needs to be improved, not about the node version to lift the maximum value.

[nazar-pc]

I've also got the same issue when compiling 4.3 MiB file with Node v8.4.0.
stderr:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x4baf13e4de [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewUninitializedFixedArray(int) [node]
 6: 0x4baec67d97 [node]
 7: 0x4baec67efc [node]
 8: 0x4baec6ad15 [node]
 9: 0x4baec6b062 [node]
10: v8::internal::Accessors::ArrayLengthSetter(v8::Local<v8::Name>, v8::Local<v8::Value>, v8::PropertyCallbackInfo<v8::Boolean> const&) [node]
11: v8::internal::PropertyCallbackArguments::Call(void (*)(v8::Local<v8::Name>, v8::Local<v8::Value>, v8::PropertyCallbackInfo<void> const&), v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>) [node]
12: v8::internal::Runtime_StoreCallbackProperty(int, v8::internal::Object**, v8::internal::Isolate*) [node]
13: 0x2bef510840dd

stdout:

<--- Last few GCs --->

[8817:0x4bb1d3f9c0]   220242 ms: Mark-sweep 1391.7 (1536.6) -> 1391.7 (1536.6) MB, 504.3 / 0.0 ms  allocation failure GC in old space requested
[8817:0x4bb1d3f9c0]   220781 ms: Mark-sweep 1391.7 (1536.6) -> 1391.7 (1477.6) MB, 539.4 / 0.0 ms  last resort 
[8817:0x4bb1d3f9c0]   221327 ms: Mark-sweep 1391.7 (1477.6) -> 1391.7 (1458.6) MB, 545.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1a71f8a9cef1 <JSObject>
    1: Djb [/media/Data/emscripten/node_modules/google-closure-compiler-js/jscomp.js:~4875] [pc=0x2bef52251a65](this=0x3a6c8af090d9 <JSGlobal Object>,a=0x1499fbbe0bd1 <Jjb map = 0x10ba718b2719>,b=0x1499fbbe09d9 <Ijb map = 0x3db5608a83d1>)
    3: rnc [/media/Data/emscripten/node_modules/google-closure-compiler-js/jscomp.js:6697] [bytecode=0xe738af9489 offset=350](this=0x3a6c8af090d9 <JSGlobal Obj...

If mentioned reproducible example is not enough, I can prepare another one.

Interestingly, this seems to happen after I've squeezed last compilation errors, it worked with incomplete externs with some complaints, but didn't crash.

[samthor]

@nazar-pc can you confirm this is happening with a large max_old_space_size value?

e.g. node --max_old_space_size 4096 ...

[nazar-pc]

@samthor, yes, node process peaked at around 4.1-4.3 GiB before it crashed.

[samthor]

What about higher—8192 etc? The Node team tell us that there should be "no limit" as of 7.3 or higher, so I'd be surprise if you couldn't get it to compile with a large number. (We're obviously not happy that so much memory is being used, but as a stop gap.)

[nazar-pc]

I was really generous, gave it 20000 MiB of memory (more than most people have in total). Node process peaked at 19.7 GiB, tried pretty hard for a few minutes, but eventually also crashed.
I think it'll eat as much memory as you allow it to.

[nazar-pc]

Here is a demo project that can be used to reproduce an issue: https://github.com/nazar-pc/closure-compiler-js-bug-23

[nazar-pc]

@samthor, is there anything else I can do to help getting this issue fixed (it is opened for almost a year now)?

[mccambridge]

I'm having the same issue. Is there any way to set this via CLI directly when I call google-closure-compiler-js, or do I have to write a script?

[nazar-pc]

@mccambridge set what? Issue is not fixed and increasing memory limit doesn't help.

[mccambridge]

@nazar-pc Gotcha. I thought I might be able to increase the limit. My project isn't huge, but it isn't small, either.

[xvilo]

I have the same issues. When compile just a simple js file it works. As soon as i include the Closure Library to it, it can't handle it :(

Edit: Upping the max limit to 8gb didn't work too!

[AntonioRedondo]

Any progress on the ticket? I came across with this issue as well, and it's a pity it prevents using this great tool.

[steveblue]

Use the Java implementation instead, it is way more stable @AntonioRedondo

[raghunathchary]

I see same issue while running ng build on docker.

92% chunk asset optimization
#
# Fatal error in , line 0
# API fatal error handler returned after process out of memory
#

[ChadKillingsworth]

For large compilations, the Java version is definitely recommended.

[AntonioRedondo]

Is this issue fixed? What happens when you have the error with small compilation setups? Is closure-compiler-js being developed actively?

[ChadKillingsworth]

Development is moving to https://github.com/google/closure-compiler-npm

If you can provide a repo case for a smaller source set, I can take a look.

[nazar-pc]

I think it is not necessarily about size, there some other issue when it uses over 20 GiB of RAM and still fails to finish compiling while Java version uses orders of magnitude less memory.

[ChadKillingsworth]

There very well may be - but I can't reproduce that. If you have a project or source set that demonstrates the problem, we can look.

[sunfeng90]

When I run webpack-dev-server --inline --progress --config build/webpack.dev.conf.js, but got `lcsdeMac-mini:jd-management-front lcs$ npm run dev

[email protected] dev /Users/lcs/Frank-Project/project/jd-project/jd-management-front
webpack-dev-server --inline --progress --config build/webpack.dev.conf.js

91% additional asset processing

Fatal error in , line 0

API fatal error handler returned after process out of memory

#FailureMessage Object: 0x70000fd71980Illegal instruction: 4`