pots left behind in case job is dying

[grembo]

In case the process of a pot dies (e.g. killall nginx) the task is restarted automatically by nomad as expected, but the pot is left behind (pot list). When doing a graceful stop of a job (nomad stop job) this isn't the case.

[pizzamig]

There is already a workaround for this, even if not fully documented.
The pots left behind can be automatically destroyed via pot prune. nomad launched pot are marked as prunable, meaning that, once stopped, they can be removed.
In minipot there is a cronjob to run pot prune periodically

[grembo]

There is already a workaround for this, even if not fully documented.
The pots left behind can be automatically destroyed via pot prune. nomad launched pot are marked as prunable, meaning that, once stopped, they can be removed.
In minipot there is a cronjob to run pot prune periodically

Is the pot marked prunable before or after launch? I'm wondering if there is a possibility for bad timing leading to accidental destruction of a pot that is in the middle of getting provisioned if pot prune is run from cron.

[pizzamig]

The mechanism is implemented in 2 steps to avoid the situation you described.
There are 2 attributes: prunable and to-be-pruned.
The attribute prunable can be set by the user and automatically set by pot prepare (this command has been designed to interact with nomad).
The attribute to-be-pruned is set by pot start if prunable is true.
pot prune will remove pots that are not running and that have the attribute prunableand to-be-pruned set to true.

[grembo]

It might make sense to integrate this feature into the pot driver (maybe configurable?).

[pizzamig]

There is a garbage collection feature in nomad and the idea is to run pot prune in that moment.
It's currently not working and we need to investigate the cause.
An alternative idea is to run pot prune after the pot is stopped.

[grembo]

@pizzamig I think the changes in bsdpot/pot#200 and #30 might actually solve this one as well.

[einsiedlerkrebs]

comment moved to #35 (as new issue)

[grembo]

I think this has been overcome by events