pm2-docker in watch and restart mode causes pm2 to exit sometimes

[dguo]

What's going wrong?

For local development, I work inside a Docker container. When I use pm2-docker with watch mode on, pm2 frequently quits after I change a file. I get the following trace:

0 application online, exiting
Exiting PM2

I did some debugging and found this commit. My guess is that there is a race condition between the setInterval in the autoExit function and the application restarting. If it doesn't restart fast enough, autoExit is triggered.

How could we reproduce this issue?

Try pm2-docker with watch mode on, and change a file repeatedly.

Supporting information

I saw the rationale for making autoExit the default for pm2-docker, but the option is still in the CLI, even though the code doesn't seem to check for it. I think either the flag should be used to turn off autoExit, or it should be removed from the CLI if it doesn't actually do anything.

In the meantime, I switched to using pm2-dev, which is working so far.

===============================================================================
--- PM2 REPORT (Tue Oct 10 2017 20:40:41 GMT+0000 (UTC)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.7.1
node version         : 8.6.0
node path            : undefined
argv                 : /usr/local/bin/node,/app/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : undefined
uid                  : 0
gid                  : 0
uptime               : 0min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.7.1
node version         : 8.6.0
node path            : undefined
argv                 : /usr/local/bin/node,/app/node_modules/.bin/pm2,report
argv0                : node
user                 : undefined
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
cpus nb              : 2
freemem              : 1549955072
totalmem             : 2096160768
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘

[RoyHP]

I am seeing the exact same behavior. Race condition; PM2 randomly exits with code 0 when the app being watched is being restarted (perhaps a check happens, seeing there's nothing running [in the middle of a restart] and PM2 exits). Naturally happens when using --auto-exit

Edit:

pm2/lib/binaries/Runtime4Docker.js

Lines 122 to 161 in 43b6000

	function autoExit() {
	var interval = 3000;
	var aliveInterval = interval * 1.5;
	setTimeout(function () {
	var alive = false
	var aliveTimer = setTimeout(function () {
	if (!alive) {
	console.error('PM2 Daemon is dead');
	process.exit(1);
	}
	}, aliveInterval);
	pm2.list(function (err, apps) {
	if (err) {
	console.log('pm2.list got error')
	console.error(err);
	exitPM2();
	}
	clearTimeout(aliveTimer);
	alive = true;
	var appOnline = 0;
	apps.forEach(function (app) {
	if (app.pm2_env.status === cst.ONLINE_STATUS ||
	app.pm2_env.status === cst.LAUNCHING_STATUS) {
	appOnline++;
	}
	});
	if (appOnline === 0) {
	console.log('0 application online, exiting');
	exitPM2();
	}
	autoExit();
	});
	}, interval);
	}

within this function, if it thinks there are 0 apps or list returns an error, it should wait a small amount of time and perform the check a second time before telling PM2 to exit. Even that could still result in the race condition, but it's less likely and I'm not familiar with PM2 enough to know if there's some way you can check to see if PM2 is in the middle of restarting something.

[Unitech]

Is anyone willing to do a small PR to fix this issue? That would be very appreciated

https://github.com/Unitech/pm2/blob/master/lib/binaries/Runtime4Docker.js

[dguo]

I can probably find some time to do it, but how would you like to fix it? Should we add a --no-auto-exit flag? The issue would still occur by default, but at least there would be a workaround.

[vmarchaud]

Since the flag is already here, you can just do something like this :

if (commander.autoExit !== false) {
 autoExit();
}

[clok]

I am seeing this race condition as well. Very interested in seeing this fixed.

[acuntex]

Seems to hang even without a watch... We currently use pm2-docker on an aws instance.
From time to time the docker container just hangs with the error message "PM2 Daemon is dead" and cannot even be killed.
Only a whole server restart fixes the issue.

[Unitech]

Fixed.

Changelog: 2.9.0 CHANGELOG
Introduced in version: 2.9.0

Update procedure:

npm install pm2 -g
pm2 update

[RoyHP]

I don't think it's safe to consider this resolved with the addition of the flag to disable auto exit - the intended functionality of auto-exit (which is useful for docker) still suffers from a race condition. Remember that this new flag is simply a workaround for the underlying issue.

Unless you mean the race condition is actually fixed in 2.9.0... I have not tested it

[erikverheij]

Although the workaround works, I can confirm the root of the issue has not been resolved as of 2.10.1.

I ran into the problem when I started using restart_delay. It seems like the auto-exit feature does not look for apps that are pending for a restart.

How to reproduce?

1. Start a single app with pm2-runtime using the following config;

    // apps.js
    module.exports = {
      apps : [{
        name: 'test',
        script: 'server.js',
        instances: 1,
        restart_delay: 3000,
      }]
    };
   

2. Simulate a crashing app in server.js

   // server.js
   setTimeout(() => {
     console.log('Simulate a crashing app');
     process.exit(0);
   }, 5000);
   

3. Run pm2-runtime config.js

• Expected behavior: The app would just restart
• Actual behavior: Pm2 exits before the app restarts

[luisdavim]

I'm having the same issue.

[quinn]

This is definitely happening

[quinn]

@Unitech Please reopen until issue is resolved

[quinn]

@Unitech to further clarify this is happening on a fresh build from keymetrics/pm2 docker image. So it's doubtful that whatever release you are referencing fixed anything.

[Unitech]

We need to be less aggressive on the shutdown:

https://github.com/Unitech/pm2/blob/master/lib/binaries/Runtime4Docker.js#L173

@quinn if you have suggestions on patching this let me know please

I think we will do 2 to 3 checks before exiting the pm2-docker.

[Unitech]

@quinn it should be good now, just update to the latest pm2:

$ npm install pm2@latest -g

Closing, if you have any issue re-open it thanks

[erikverheij]

Thanks for the update.

I don't think the fix is really solving the underlying problem. It now waits a bit longer, but whenever you've configured restart_delay > 4000 it will still exit the app while it shouldn't.

Steps to reproduce;

1. Start a single app with pm2-runtime using the following config;

    // apps.config.js
    module.exports = {
      apps : [{
        name: 'test',
        script: 'server.js',
        instances: 1,
        restart_delay: 5000,
      }]
    };
   

2. Simulate a crashing app in server.js

   // server.js
   setTimeout(() => {
     console.log('Simulate a crashing app');
     process.exit(0);
   }, 5000);
   

3. Run pm2-runtime apps.config.js

• Expected behavior: The app would just restart
• Actual behavior: Pm2 exits before the app restarts

Would it be an idea to to add cst.LAUNCH_PENDING_STATUS? Or another solution might be that the autoexit timeout is configurable. If you want a long time between restarts you can also set a long timeout for autoexit.