Feature request: exec a command

[lsorber]

When using Poe the Poet as an entrypoint in a Docker image, it would be nice to be able to exec [1] commands. For example: exec gunicorn my_service.api:app. In Python, this can be achieved with os.execvp [2].

[1] https://man7.org/linux/man-pages/man3/exec.3.html
[2] https://docs.python.org/3/library/os.html#os.execvp

[nat-n]

Hi @lsorber, I'd be interested to learn more about your use case. How/why/what for are you using poe in docker?

This is actually how poe worked in early versions, in that it would essentially replace the current process with the target task. However I switched to process forking via subprocess.Popen in order to support running multiple tasks in one go and simplify cross platform support.

Maybe we could better support your use case if you elaborate on what you're trying to do?

[lsorber]

Poe the Poet is great for running tasks locally, such as serving an API with something like poe serve. When packaging a service in a Docker image, ideally you'd run exactly the same task to minimise potential differences between how you serve the API locally and how you serve it with Docker. In other words, you want docker run to invoke the same Poe the Poet task you run locally to run your application.

To create a Docker image that runs a Poe the Poet task, you can set the final ENTRYPOINT instruction to use poe and set the CMD instruction to the task you want to run:

ENTRYPOINT ["/usr/local/bin/poe"]
CMD ["serve"]

However, if the entrypoint is not the process you actually intend to run (as is the case here), then Docker recommends execing the underlying process to allow it to receive any Unix signals sent to the container [1]. See also this SO answer for more details on why this is important [2].

[1] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#entrypoint
[2] https://stackoverflow.com/a/48096779/16086151

[nat-n]

I wouldn't rule out adding the ability to make poe execv a specific task instead of running it as a subprocess, though it would introduce a bit of complexity (e.g. special casing for certain task types, or any kind of task composition), hence there is some judgement to be made on whether it's worth living with that complexity.

As far as handling of signals, poe intends to do the right thing in terms of forwarding signals to the current task. If there's something important missing here we should probably improve it.

For the sake of discussion: poethepoet is intentionally quite light, though unless what you want is a really general purpose docker image you might not want to have to ship poe (or poetry?) or anything else in your docker image that isn't essential for running your service. Alternative approaches might include:

1. simply duplicate some contents in both the poe task in pyproject and the Dockerfile (or have task to sync them)
2. run docker from poe, maybe as a distinct serve-prod task (I've done it this way myself before)

[lsorber]

For additional context, we've open-sourced a Poetry Cookiecutter template that makes use of Poe the Poet.

The template has a Docker-based development environment (with VS Code's Dev Containers and PyCharm's Docker Compose interpreter) so that it's easy to start contributing and so that differences between the local and the deployed service are minimised. And since we're using Poe to run tasks such as serving an API locally (e.g. with poe serve), to us it makes sense to also use poe as the Dockerfile's entrypoint to start serving exactly the same way when the service is deployed (e.g. with docker run my-service serve).

That being said, I haven't experienced any actual issues with signals not being forwarded by Poe to the underlying process yet, but it seems a bit safer to not have to have that layer around the command by exec'ing it. Would it make sense to exec only the last command in an array?

[nat-n]

Nice project 👍

Given that poe supports composing tasks in multiple ways, (and I don't want to close off other possibilities that are internally supported but not yet implemented) – there are a bunch of potentially tricky edge cases for automatically deciding whether to use a forked subprocess or exec within the current process.

However, what would lower the stakes a bit would be to make it an opt-in configuration along the lines of:

[tool.poe.tasks.serve]
cmd = "gunicorn ./my_app:run"
use_exec = true

To simplify things a task with use_exec might not be allowed to be composed with or referenced by another task.

Would this be satisfactory?

[lsorber]

I think that would be a pretty nice feature, yes!

[nat-n]

This feature is now available in the v0.15.0