add pip-compatible `--group` flag to `uv pip install` and `uv pip compile` #11686

Gankra · 2025-02-21T05:02:32Z

This is a minimal redux of #10861 to be compatible with uv pip.

This implements the interface described in: pypa/pip#13065 (comment) for uv pip install and uv pip compile. Namely --group <[path:]name>, where path when not defined defaults to pyproject.toml.

In that interface they add --group to pip install, pip download, and pip wheel. Notably we do not define uv pip download and uv pip wheel, so for parity we only need to implement uv pip install. However, we also support uv pip compile which is not part of pip itself, and --group makes sense there too.

The behaviour of --group for uv pip commands makes sense for the cases upstream pip supports, but has confusing meanings in cases that only we support (because reading pyproject.tomls is New Tech to them but heavily supported by us). Specifically case (h) below is a concerning footgun, and case (e) below may get complaints from people who aren't well-versed in dependency-groups-as-they-pertain-to-wheels.

Only Group Flags

Group flags on their own work reasonably and uncontroversially, except perhaps that they don't do very clever automatic project discovery.

a) uv pip install --group path/to/pyproject.toml:mygroup pulls up path/to/project.toml and installs all the packages listed by its mygroup dependency-group (essentially treating it like another kind of requirements.txt). In this regard it functions similarly to --only-group in the rest of uv's interface.

b) uv pip install --group mygroup is just sugar for uv pip install --group pyproject.toml:mygroup (note that no project discovery occurs, upstream pip simply hardcodes the path "pyproject.toml" here and we reproduce that.)

c) uv pip install --group a/pyproject.toml:groupx --group b/pyproject.toml:groupy, and any other instance of multiple --group flags, can be understood as completely independent requests for the given groups at the given files.

Groups With Named Packages

Groups being mixed with named packages also work in a fairly unsurprising way, especially if you understand that things like dependency-groups are not really supposed to exist on pypi, they're just for local development.

d) uv pip install mypackage --group path/to/pyproject.toml:mygroup much like multiple instances of --group the two requests here are essentially completely independent: pleases install mypackage, and please also install path/to/pyproject.toml:mygroup.

e) uv pip install mypackage --group mygroup is exactly the same, but this is where it becomes possible for someone to be a little confused, as you might think mygroup is supposed to refer to mypackage in some way (it can't). But no, it's sourcing pyproject.toml:mygroup from the current working directory.

Groups With Requirements/Sourcetrees/Editables

Requirements and sourcetrees are where I expect users to get confused. It behaves exactly the same as it does in the previous sections but you would absolutely be forgiven for expecting a different behaviour. Especially because --group with the rest of uv does do something different.

f) uv pip install -r a/pyproject.toml --group b/pyproject.toml:mygroup is again just two independent requests (install a/pyproject.toml's dependencies, and b/pyproject.toml's mygroup).

g) uv pip install -r pyproject.toml --group mygroup is exactly like the previous case but incidentally the two requests refer to the same file. What the user wanted to happen is almost certainly happening, but they are likely getting "lucky" here that they're requesting something simple.

h) uv pip install -r a/pyproject.toml --group mygroup is again exactly the same but the user is likely to get surprised and upset as this invocation actually sources two different files (install a/pyproject.toml's dependencies, and pyproject.toml's mygroup)! I would expect most people to assume the --group flag here is covering all applicable requirements/sourcetrees/editables, but no, it continues to be a totally independent reference to a file with a hardcoded relative path.

Fixes #8590
Fixes #8969

Gankra · 2025-02-21T15:35:54Z

crates/uv/src/commands/pip/operations.rs

+        // Deduplicate in a stable way to get deterministic behaviour
+        let deduped_paths = groups
+            .iter()
+            .map(|group| &group.path)
+            .collect::<BTreeSet<_>>();
+        group_requirements = deduped_paths
+            .into_iter()
+            .map(|path| RequirementsSource::PyprojectToml(path.to_owned()))
+            .collect::<Vec<_>>();
+        requirements = &group_requirements[..];


I started doing a cleanup pass to compute a BTreeMap<PathBuf, Vec<GroupName>> that would be used by subsequent steps but it was just more code and value threading for no real benefit over the current impl that just does a filter(group.path == path) later.

henryiii · 2025-02-23T14:25:57Z

FYI, pip support was merged.

zanieb · 2025-02-24T20:05:55Z

crates/uv-cli/src/lib.rs

+    /// Ignore the package and it's dependencies, only install from the specified dependency group.
+    ///
+    /// May be provided multiple times.
+    #[arg(long, group = "sources", conflicts_with_all = ["requirements", "package", "editable"])]


Are these conflicts present in the pip implementation?

(it doesn't really look like it?)

I thought -r was exclusive to our CLI (I didn't check the others though).

Yeah -r is exclusive to us, but I'd expect pip install -e . --group foo and pip install anyio --group foo to work unless they explicitly choose for it not to upstream.

(Whether -r should have the same semantics as these other options, I'm not sure)

One problem is that uv pip install -r pyproject.toml --extra foo is the "correct" use of --extra but uv pip install -r pyproject.toml --group foo would be invalid? Even more confusing, if we allow it, uv pip install -r foo/pyproject.toml --group bar would install bar from ./pyproject.toml instead of foo/pyproject.toml.

Yeah that's part of why I opted to disallow it. The upstream pip semantics are really confusing to pair with anything else.

I think it's important that we're compatible with pip on whether --group can be used with <package> and --editable though, which introduces all the same complexities.

Gankra · 2025-02-27T04:53:05Z

I implemented/pushed the following changes, because I needed to wrap my head around them in code/tests to think them through semantically:

totally unrestrict --group from all other sources in uv-pip-install
- semantically i still have it that --group just installs that group in a pyproject.toml (still hardcoded ./pyproject.toml default for now, see below). All the following cases are accepted:
  - "unrelated": uv pip install somepackage --group bar (essentially sugar for two separate uv pip install invocations as it is in upstream pip install)
  - "nice": uv pip install -r pyproject.toml --group bar (installs both the package's deps and its group bar) (but this is the user of the cli potentially getting "lucky")
  - "confusing": uv pip install -r foo/project.toml --group bar (installs foo/pyproject.toml's deps and ./pyproject.toml:bar)
- implementation-wise this was a bit hairy as we have to kind of detect if we're doing --group or --only-group by looking at all the other inputs, but it ends up making sense (really it's always --only-group-but-additive-only... it's just an artifact of the internal source_tree stuff and wanting to dedupe files that we need to do anything complex).
enable the same semantics for uv-pip-compile
- ...this one I "implemented" and then I went to test it and realized that the semantics for pip-install are absolute nonsense here, i think? i think the old-old-old impl from last month where the --group just applies to every selected source is what you want here..? I'm not totally sure, I don't have a good intuition of what uses of this would look like... but I get the impression that uv pip compile --group pyproject.toml:foo is essentially gibberish? Or am I wrong?
did not add project discovery in lieu of hardcoding pyproject.toml -- I was going to do this but then I realized that nowhere in the pip interface do we ever seem to invoke project discovery? Which was surprising. This seems like a big semantic departure if we start doing that? At very least needs to be discussed more.

Gankra · 2025-02-27T15:59:25Z

Zanie convinced me that pip-compile is fine actually, so I just loosened the CLI restriction to make it accepted.

I'm just adding more tests now.

zanieb · 2025-03-03T20:21:37Z

Can you share the current state of this? It looks like you updated the PR summary to cover the behavior we discussed?

Gankra · 2025-03-04T03:46:53Z

I believe the PR is in the final state I'd like. Possibly it could have Even More tests but I'm not sure it would add much. (Just rebased)

…pile`

zanieb

This looks pretty good. I have mostly minor comments, though the --project bit might be blocking?

We should add this to the documentation at

We may want to mark --group on uv pip compile as "in preview" until support lands in pip-tools. See jazzband/pip-tools#2062. It looks like they're just considering using :: instead of : as a separator? Moving this to preview would involve a runtime warning and a note in the doc.

crates/uv/tests/it/pip_install.rs

crates/uv-cli/src/lib.rs

crates/uv-normalize/src/group_name.rs

crates/uv/tests/it/pip_install.rs

crates/uv-cli/src/lib.rs

crates/uv-requirements/src/specification.rs

crates/uv/tests/it/pip_install.rs

zanieb · 2025-03-04T21:15:50Z

crates/uv-requirements/src/specification.rs

+            requirement_sources.push(source);
+        }
+
+        // pip `--group` flags specify their own sources, which we need to process here


I'm pretty sure this makes sense, but it might be relevant for Charlie to take a look at before merging.

Gankra · 2025-03-08T00:42:18Z

Ooookay I think that's all review feedback addressed!

zanieb · 2025-03-10T19:19:27Z

docs/pip/compile.md

+```console
+$ uv pip compile --group some/path/pyproject.toml:foo
+```
+


We should also mention --project here, especially for multiple groups.

We may want to add a !!! note admonition noting that this is not yet implemented in pip-tools yet and link to the issue?

zanieb · 2025-03-10T19:20:31Z

docs/pip/compile.md

+To lock dependency groups in the current project directory's pyproject.toml:
+
+```console
+$ uv pip compile --group foo
+```


"To lock a dependency group... For example, to lock the group foo"?

We also use backticks for pyproject.toml.

zanieb · 2025-03-10T19:21:02Z

docs/pip/packages.md

@@ -107,6 +107,18 @@ Install from a `pyproject.toml` file with all optional dependencies enabled:
 $ uv pip install -r pyproject.toml --all-extras
 ```

+To install dependency groups in the current project directory's pyproject.toml:


My above comments for pip compile apply here too.

zanieb · 2025-03-10T19:21:58Z

docs/pip/packages.md

+To install dependency groups for an arbitrary pyproject.toml:
+
+```console
+$ uv pip install --group some/path/pyproject.toml:foo
+```


We need to talk about how --group does not apply to other sources, like -e . etc. here. Maybe under an !!! note admonition? We could track this separately, but this is an important thing to teach in the docs.

zanieb · 2025-03-10T19:23:03Z

crates/uv/tests/it/pip_install.rs

+    ----- stdout -----
+
+    ----- stderr -----
+    error: invalid value './:foo' for '--group <GROUP>': The --group path is required to end in pyproject.toml for compatibility with pip; got: ./


Should we quote 'pyproject.toml' for consistency with the rest?

zanieb · 2025-03-10T19:23:48Z

crates/uv/tests/it/pip_install.rs

+    ----- stdout -----
+
+    ----- stderr -----
+    error: invalid value 'subdir/:foo' for '--group <GROUP>': The --group path is required to end in pyproject.toml for compatibility with pip; got: subdir/


--group should be wrapped in backticks

zanieb · 2025-03-10T19:24:13Z

crates/uv/tests/it/pip_install.rs

+    ----- stdout -----
+
+    ----- stderr -----
+    error: The dependency group 'bar' was not found in pyproject.toml


Probably want "in the pyproject.toml"

zanieb

I think this is ready. It's hard to keep track of everything with the scope of the changes.

I have a few stylistic nits remaining and want to see more user-facing documentation explaining how --group interacts with other options and how it differs from the rest of the uv interface. We can push that into a separate pull request, but it's very important. I'm happy to help with that if you need it.

Gankra commented Feb 21, 2025

View reviewed changes

zanieb reviewed Feb 24, 2025

View reviewed changes

Gankra force-pushed the gankra/pip-groups-again branch from 6e578d9 to 9376d4f Compare February 27, 2025 16:04

Gankra changed the title ~~add pip-compatible --group flag to uv pip install~~ add pip-compatible --group flag to uv pip install and uv pip compile Feb 27, 2025

davfsa mentioned this pull request Mar 3, 2025

Configure Renovate hikari-py/hikari#2184

Merged

Gankra force-pushed the gankra/pip-groups-again branch from 9376d4f to 4d3b843 Compare March 4, 2025 03:44

Gankra force-pushed the gankra/pip-groups-again branch from 4d3b843 to 1cc738b Compare March 4, 2025 04:00

add pip-compatible --group flag to uv pip install and `uv pip com…

0dfe672

…pile`

Gankra force-pushed the gankra/pip-groups-again branch from 1cc738b to 0dfe672 Compare March 4, 2025 04:05

zanieb mentioned this pull request Mar 4, 2025

Eventually implement support for the PEP 735 "dependency groups" jazzband/pip-tools#2062

Open

zanieb reviewed Mar 4, 2025

View reviewed changes

Gankra added 6 commits March 6, 2025 14:07

address some feedback

d59a8bf

add new requirement origin for dependency groups

4040c85

handle --project

028928a

add a ton of tests for --groups

c9af618

generate

4bcdb26

update docs

36ebee3

zanieb reviewed Mar 10, 2025

View reviewed changes

zanieb approved these changes Mar 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add pip-compatible `--group` flag to `uv pip install` and `uv pip compile` #11686

add pip-compatible `--group` flag to `uv pip install` and `uv pip compile` #11686

Gankra commented Feb 21, 2025 •

edited

Loading

Gankra Feb 21, 2025

henryiii commented Feb 23, 2025

zanieb Feb 24, 2025

zanieb Feb 24, 2025

Gankra Feb 24, 2025

zanieb Feb 24, 2025

zanieb Feb 24, 2025 •

edited

Loading

zanieb Feb 24, 2025 •

edited

Loading

Gankra Feb 24, 2025

zanieb Feb 24, 2025

Gankra commented Feb 27, 2025 •

edited

Loading

Gankra commented Feb 27, 2025

zanieb commented Mar 3, 2025

Gankra commented Mar 4, 2025

zanieb left a comment •

edited

Loading

zanieb Mar 4, 2025

Gankra commented Mar 8, 2025

zanieb Mar 10, 2025

zanieb Mar 10, 2025

zanieb Mar 10, 2025 •

edited

Loading

zanieb Mar 10, 2025

zanieb Mar 10, 2025

zanieb Mar 10, 2025

zanieb Mar 10, 2025

zanieb Mar 10, 2025

zanieb left a comment

add pip-compatible --group flag to uv pip install and uv pip compile #11686

Are you sure you want to change the base?

add pip-compatible --group flag to uv pip install and uv pip compile #11686

Conversation

Gankra commented Feb 21, 2025 • edited Loading

Only Group Flags

Groups With Named Packages

Groups With Requirements/Sourcetrees/Editables

Choose a reason for hiding this comment

henryiii commented Feb 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

zanieb Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gankra commented Feb 27, 2025 • edited Loading

Gankra commented Feb 27, 2025

zanieb commented Mar 3, 2025

Gankra commented Mar 4, 2025

zanieb left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gankra commented Mar 8, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb Mar 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb left a comment

Choose a reason for hiding this comment

add pip-compatible `--group` flag to `uv pip install` and `uv pip compile` #11686

add pip-compatible `--group` flag to `uv pip install` and `uv pip compile` #11686

Gankra commented Feb 21, 2025 •

edited

Loading

zanieb Feb 24, 2025 •

edited

Loading

zanieb Feb 24, 2025 •

edited

Loading

Gankra commented Feb 27, 2025 •

edited

Loading

zanieb left a comment •

edited

Loading

zanieb Mar 10, 2025 •

edited

Loading