Include build dependencies in locked resolutions (uv pip compile, uv lock)

[alex]

Right now uv pip compile does not include build dependencies in its output. It would be incredibly helpful if there was a way to include them.

[charliermarsh]

👍 Can you walk me through how you'd use this? Like, if they were included in the requirements.txt, how would you "restore" / respect them in subsequent commands given build isolation?

[charliermarsh]

(I want to support this.)

[alex]

My immediate use case is to prepare constraints files that can be used with --build-constraints

[alex]

Now that uv build has --build-constraint support, this would enable easily building those constraints files. Right now we do it by manually preparing a requirements.in from the build-system.requires, but that's obviously manual and error prone.

[charliermarsh]

What if the build requirements conflict between different packages though?

[alex]

Interesting question, I hadn't considered that. I'm not sure that would occur in our use case (taking specific local projects and getting their build dependencies), but clearly in the general case, it'd be a problem.

[lmmx]

If I'm understanding this rightly, then this is also desirable for building wheels in a Docker container from source, in which situation you want to programmatically obtain the build requirements before doing a --no-build-isolation run of pip download. You may assume here they are compatible, and just need resolving (i.e. a uv pip compile step), without this you just do this by hand by inspecting the build-system.requires TOML section of each package's build dependencies).

In this case conflict would be fine to error out on, it'd just cue the user to split this download process up per-requirement (or use separate lists?

I've ended up making distinct subpackages named (dependencyname_build) with the build system requirements as dependencies in my current project to achieve this... (WIP)

[notatallshaw]

Here's my 2 cents:

1. Unlike pip-compile which includes build dependencies in it's regular output with --all-build-deps, uv pip compile should only have an option to output build dependencies only, this is because uv supports --build-constraints where as pip-tools is bound by the fact pip can only pass build constraints via PIP_CONSTRAINT, which also affects runtime dependencies
2. It should be recommended / defaulted / forced to use --only-binary :all:, for the build dependencies, when resolving build dependencies, as not to deal with recursive build dependencies
3. It should be documented that if the user wants maximum reproducibility they must create the build dependencies of each runtime requirement source distribution, and build each one with it's own build constraints
4. If there is a conflict in build dependencies point them to some version of 3, that they must build the packages independently
5. It might make sense not to resolve runtime requirements when resolving build dependencies, so the workflow would be requirement.in -> requirements.txt -> build-constraints.txt, I think you would need to consider carefully if all existing CLI options apply to runtime requirements or build dependencies or both

Build dependencies are complicated enough that you will not be able to capture all use cases in a requirements file, but this approach should be "good enough" for a lot of users I think. Most users will probably find they are pinned to some versions of setuptools and some version of hatch, and maybe one or two more build backends, and that's it.

However, for true locking you would need to be able to capture the potential recursive nature of build dependencies in a lock file format. Unfortunately when I asked some questions about PEP 751 build dependencies the response was to drop build dependencies from the spec.

Finally these would be "nice to have" features if this uv pip compile "build-dependencies" options existed:

1. Generate build dependencies for all the packages in a requirement file and it only generate the requirements for those packages that are "allowed" to be sdists, taking into account --only-binary and --no-binary options (although again, this is complex on what the CLI options are being applied to)
2. The hash output should (optionally) not include sdist hashes

P.S. I'm sure there's stuff I'm not thinking of, e.g. maybe it makes more sense for uv that the build constraints are outputed into a seperate output file instead of only outputing the build constraints for various technical or workflow reasons.

[alex]

Doing build deps only would be fine for me

…

On Tue, Sep 17, 2024, 12:34 PM Damian Shaw ***@***.***> wrote: Here's my 2 cents: 1. Unlike pip-compile which includes build dependencies in it's regular output <https://github.com/jazzband/pip-tools?tab=readme-ov-file#maximizing-reproducibility> with --all-build-deps, uv pip compile should only have an option to output build dependencies only, this is because uv supports --build-constraints where as pip-tools is bound by the fact pip can only pass build constraints via PIP_CONSTRAINT 2. It should be recommended / defaulted / forced to use --only-binary :all: when resolving build dependencies, as not to deal with recursive build requirements 3. It should be documented that if the user wants maximum reproducibility they must create the build requirements of each runtime requirement source distribution, and build each one with it's own build constraints 4. If there is a conflict in build requirements point them to some version of 3, that they must build the packages independently Build dependencies are complicated enough that you will not be able to capture all use cases in a requirements file, but this approach should be "good enough" for a lot of users I think. Most users will probably find they are pinned to some versions of setuptools and some version of hatch, and maybe one or two more build backends, and that's it. However, for true locking you would need to be able to capture the potential recursive nature of build requirements in a lock file format. Unfortunately when I asked some questions anout PEP 751 the response was to drop build requirements <https://discuss.python.org/t/pep-751-lock-files-again/59173/222> from the spec. Finally these would be "nice to have" features if this uv pip compile "build-requirements" options existed: 1. Generate build requirements for all the packages in a requirement file and it only generate the requirements for those packages that are "allowed" to be sdists, taking into account --only-binary and --no-binary options. 2. The hash output not include sdist hashes — Reply to this email directly, view it on GitHub <#7052 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBDPNF3AMPKAKWE7W73ZXBK25AVCNFSM6AAAAABNVFZFTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJWGQYDMNZQHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[charliermarsh]

Yeah the thing I don't know how to solve is (4) -- that every dependency defines its own build requirements, and they're welcome to conflict.

[notatallshaw]

Yeah the thing I don't know how to solve is (4) -- that every dependency defines its own build requirements, and they're welcome to conflict.

For producing a constraints file (and not a structured lock file) I don't think it's possible.

The user would be responsible for handling this situation themselves, they would need to build each package separately with their own file, e.g by making a script that loops over their requirements and running like this overly simplified pseudo code:

for requirement in sdist-requirements.txt:
	echo {requirement} | uv pip compile {build dependency options} - > build-constraints-{requirement}.txt
	uv build {requirement} --build-constraints build-constraints-{requirement}.txt
	{move wheel to index / file location}
uv pip install {install options that include newly built wheel}

However, I suspect this would affect a very small number of users.

[alex]

I guess my answer would be: within the limits of the requirements.txt format, you probably can't solve it fully. But then what Damian says comes in: it effects probably a small number of users, and just unifying all of them likely works well enough, and doesn't precludes a better format later.

…

On Tue, Sep 17, 2024, 1:56 PM Charlie Marsh ***@***.***> wrote: Yeah the thing I don't know how to solve is (4) -- that every dependency defines its own build requirements, and they're welcome to conflict. — Reply to this email directly, view it on GitHub <#7052 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBGUEM6RINC5YRBHS2TZXBULXAVCNFSM6AAAAABNVFZFTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJWGU3DIMBTGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alex]

Is this still blocked on a design decision for how to represent build deps in requirements.txt files, or are there other issues here?

[charliermarsh]

We could, like, output a build constraints file that includes all the constraints required for the all build dependencies, assuming they aren't in conflict?

[charliermarsh]

I think this is mostly blocked on us prioritizing it. I was sort of holding off while PEP 751 was in flight, but I have more clarity on that now.

[alex]

We could, like, output a build constraints file that includes all the constraints required for the all build dependencies, assuming they aren't in conflict?

Personally that works for my use case (where I just want to generate a lockfile for one package's build deps).

Cheers

[neutrinoceros]

I feel like this would generally be useful if it can be sorted out, but I'd like to illustrate one specific use case where it could be argued that this feature is actually vital:
Say I develop a package (pkg) whose build backend doesn't support build isolation¹. I can specify

[tool.uv]
no-build-isolation-package = ["pkg"]

which forbids building in isolation, but does still requires me to install my (unresolved) build-time dependencies manually somehow before I can build pkg. If build-time dependencies are not locked or included automatically, this means I essentially
cannot use high level APIs like uv sync and have to resort to uv pip install. My current workaround is to duplicate build-time requirements into the dev dependency group so they are always installed on uv sync (unless specifically opted-out), but this is fragile for two reasons:

• I now need to manually keep [build-system] and [dependency-groups] in sync
• I still can't bootstrap my env in a single command. Rather, I need to do uv sync --only-dev && uv sync, which feels like a hack (which it is, I suppose).

Footnotes

1. I'm simplifying a bit, but the real-life use case is that I use meson-python as a build backend, which by design, supports build isolation or editable installs, but not the combination of both. This is done because C-extensions installed in editable mode require re-compilation ahead of runtime, which can only be performed if build-time requirements are available in the same environment. ↩