Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add pip-compatible --group flag to uv pip install and uv pip compile #11686

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

Gankra
Copy link
Contributor

@Gankra Gankra commented Feb 21, 2025

This is a minimal redux of #10861 to be compatible with uv pip.

This implements the interface described in: pypa/pip#13065 (comment) for uv pip install and uv pip compile. Namely --group <[path:]name>, where path when not defined defaults to pyproject.toml.

In that interface they add --group to pip install, pip download, and pip wheel. Notably we do not define uv pip download and uv pip wheel, so for parity we only need to implement uv pip install. However, we also support uv pip compile which is not part of pip itself, and --group makes sense there too.


The behaviour of --group for uv pip commands makes sense for the cases upstream pip supports, but has confusing meanings in cases that only we support (because reading pyproject.tomls is New Tech to them but heavily supported by us). Specifically case (h) below is a concerning footgun, and case (e) below may get complaints from people who aren't well-versed in dependency-groups-as-they-pertain-to-wheels.

Only Group Flags

Group flags on their own work reasonably and uncontroversially, except perhaps that they don't do very clever automatic project discovery.

a) uv pip install --group path/to/pyproject.toml:mygroup pulls up path/to/project.toml and installs all the packages listed by its mygroup dependency-group (essentially treating it like another kind of requirements.txt). In this regard it functions similarly to --only-group in the rest of uv's interface.

b) uv pip install --group mygroup is just sugar for uv pip install --group pyproject.toml:mygroup (note that no project discovery occurs, upstream pip simply hardcodes the path "pyproject.toml" here and we reproduce that.)

c) uv pip install --group a/pyproject.toml:groupx --group b/pyproject.toml:groupy, and any other instance of multiple --group flags, can be understood as completely independent requests for the given groups at the given files.

Groups With Named Packages

Groups being mixed with named packages also work in a fairly unsurprising way, especially if you understand that things like dependency-groups are not really supposed to exist on pypi, they're just for local development.

d) uv pip install mypackage --group path/to/pyproject.toml:mygroup much like multiple instances of --group the two requests here are essentially completely independent: pleases install mypackage, and please also install path/to/pyproject.toml:mygroup.

e) uv pip install mypackage --group mygroup is exactly the same, but this is where it becomes possible for someone to be a little confused, as you might think mygroup is supposed to refer to mypackage in some way (it can't). But no, it's sourcing pyproject.toml:mygroup from the current working directory.

Groups With Requirements/Sourcetrees/Editables

Requirements and sourcetrees are where I expect users to get confused. It behaves exactly the same as it does in the previous sections but you would absolutely be forgiven for expecting a different behaviour. Especially because --group with the rest of uv does do something different.

f) uv pip install -r a/pyproject.toml --group b/pyproject.toml:mygroup is again just two independent requests (install a/pyproject.toml's dependencies, and b/pyproject.toml's mygroup).

g) uv pip install -r pyproject.toml --group mygroup is exactly like the previous case but incidentally the two requests refer to the same file. What the user wanted to happen is almost certainly happening, but they are likely getting "lucky" here that they're requesting something simple.

h) uv pip install -r a/pyproject.toml --group mygroup is again exactly the same but the user is likely to get surprised and upset as this invocation actually sources two different files (install a/pyproject.toml's dependencies, and pyproject.toml's mygroup)! I would expect most people to assume the --group flag here is covering all applicable requirements/sourcetrees/editables, but no, it continues to be a totally independent reference to a file with a hardcoded relative path.


Fixes #8590
Fixes #8969

Comment on lines 68 to 77
// Deduplicate in a stable way to get deterministic behaviour
let deduped_paths = groups
.iter()
.map(|group| &group.path)
.collect::<BTreeSet<_>>();
group_requirements = deduped_paths
.into_iter()
.map(|path| RequirementsSource::PyprojectToml(path.to_owned()))
.collect::<Vec<_>>();
requirements = &group_requirements[..];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started doing a cleanup pass to compute a BTreeMap<PathBuf, Vec<GroupName>> that would be used by subsequent steps but it was just more code and value threading for no real benefit over the current impl that just does a filter(group.path == path) later.

@henryiii
Copy link
Contributor

FYI, pip support was merged.

/// Ignore the package and it's dependencies, only install from the specified dependency group.
///
/// May be provided multiple times.
#[arg(long, group = "sources", conflicts_with_all = ["requirements", "package", "editable"])]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these conflicts present in the pip implementation?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(it doesn't really look like it?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought -r was exclusive to our CLI (I didn't check the others though).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah -r is exclusive to us, but I'd expect pip install -e . --group foo and pip install anyio --group foo to work unless they explicitly choose for it not to upstream.

Copy link
Member

@zanieb zanieb Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Whether -r should have the same semantics as these other options, I'm not sure)

Copy link
Member

@zanieb zanieb Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One problem is that uv pip install -r pyproject.toml --extra foo is the "correct" use of --extra but uv pip install -r pyproject.toml --group foo would be invalid? Even more confusing, if we allow it, uv pip install -r foo/pyproject.toml --group bar would install bar from ./pyproject.toml instead of foo/pyproject.toml.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's part of why I opted to disallow it. The upstream pip semantics are really confusing to pair with anything else.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's important that we're compatible with pip on whether --group can be used with <package> and --editable though, which introduces all the same complexities.

@Gankra
Copy link
Contributor Author

Gankra commented Feb 27, 2025

I implemented/pushed the following changes, because I needed to wrap my head around them in code/tests to think them through semantically:

  • totally unrestrict --group from all other sources in uv-pip-install
    • semantically i still have it that --group just installs that group in a pyproject.toml (still hardcoded ./pyproject.toml default for now, see below). All the following cases are accepted:
      • "unrelated": uv pip install somepackage --group bar (essentially sugar for two separate uv pip install invocations as it is in upstream pip install)
      • "nice": uv pip install -r pyproject.toml --group bar (installs both the package's deps and its group bar) (but this is the user of the cli potentially getting "lucky")
      • "confusing": uv pip install -r foo/project.toml --group bar (installs foo/pyproject.toml's deps and ./pyproject.toml:bar)
    • implementation-wise this was a bit hairy as we have to kind of detect if we're doing --group or --only-group by looking at all the other inputs, but it ends up making sense (really it's always --only-group-but-additive-only... it's just an artifact of the internal source_tree stuff and wanting to dedupe files that we need to do anything complex).
  • enable the same semantics for uv-pip-compile
    • ...this one I "implemented" and then I went to test it and realized that the semantics for pip-install are absolute nonsense here, i think? i think the old-old-old impl from last month where the --group just applies to every selected source is what you want here..? I'm not totally sure, I don't have a good intuition of what uses of this would look like... but I get the impression that uv pip compile --group pyproject.toml:foo is essentially gibberish? Or am I wrong?
  • did not add project discovery in lieu of hardcoding pyproject.toml -- I was going to do this but then I realized that nowhere in the pip interface do we ever seem to invoke project discovery? Which was surprising. This seems like a big semantic departure if we start doing that? At very least needs to be discussed more.

@Gankra
Copy link
Contributor Author

Gankra commented Feb 27, 2025

Zanie convinced me that pip-compile is fine actually, so I just loosened the CLI restriction to make it accepted.

I'm just adding more tests now.

@Gankra Gankra force-pushed the gankra/pip-groups-again branch from 6e578d9 to 9376d4f Compare February 27, 2025 16:04
@Gankra Gankra changed the title add pip-compatible --group flag to uv pip install add pip-compatible --group flag to uv pip install and uv pip compile Feb 27, 2025
@zanieb
Copy link
Member

zanieb commented Mar 3, 2025

Can you share the current state of this? It looks like you updated the PR summary to cover the behavior we discussed?

@Gankra Gankra force-pushed the gankra/pip-groups-again branch from 9376d4f to 4d3b843 Compare March 4, 2025 03:44
@Gankra
Copy link
Contributor Author

Gankra commented Mar 4, 2025

I believe the PR is in the final state I'd like. Possibly it could have Even More tests but I'm not sure it would add much. (Just rebased)

@Gankra Gankra force-pushed the gankra/pip-groups-again branch from 4d3b843 to 1cc738b Compare March 4, 2025 04:00
Copy link
Member

@zanieb zanieb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good. I have mostly minor comments, though the --project bit might be blocking?

We should add this to the documentation at

We may want to mark --group on uv pip compile as "in preview" until support lands in pip-tools. See jazzband/pip-tools#2062. It looks like they're just considering using :: instead of : as a separator? Moving this to preview would involve a runtime warning and a note in the doc.

requirement_sources.push(source);
}

// pip `--group` flags specify their own sources, which we need to process here
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure this makes sense, but it might be relevant for Charlie to take a look at before merging.

@Gankra
Copy link
Contributor Author

Gankra commented Mar 8, 2025

Ooookay I think that's all review feedback addressed!

```console
$ uv pip compile --group some/path/pyproject.toml:foo
```

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also mention --project here, especially for multiple groups.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to add a !!! note admonition noting that this is not yet implemented in pip-tools yet and link to the issue?

Comment on lines +63 to +67
To lock dependency groups in the current project directory's pyproject.toml:

```console
$ uv pip compile --group foo
```
Copy link
Member

@zanieb zanieb Mar 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"To lock a dependency group... For example, to lock the group foo"?

We also use backticks for pyproject.toml.

@@ -107,6 +107,18 @@ Install from a `pyproject.toml` file with all optional dependencies enabled:
$ uv pip install -r pyproject.toml --all-extras
```

To install dependency groups in the current project directory's pyproject.toml:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My above comments for pip compile apply here too.

Comment on lines +116 to +120
To install dependency groups for an arbitrary pyproject.toml:

```console
$ uv pip install --group some/path/pyproject.toml:foo
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to talk about how --group does not apply to other sources, like -e . etc. here. Maybe under an !!! note admonition? We could track this separately, but this is an important thing to teach in the docs.

----- stdout -----

----- stderr -----
error: invalid value './:foo' for '--group <GROUP>': The --group path is required to end in pyproject.toml for compatibility with pip; got: ./
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we quote 'pyproject.toml' for consistency with the rest?

----- stdout -----

----- stderr -----
error: invalid value 'subdir/:foo' for '--group <GROUP>': The --group path is required to end in pyproject.toml for compatibility with pip; got: subdir/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--group should be wrapped in backticks

----- stdout -----

----- stderr -----
error: The dependency group 'bar' was not found in pyproject.toml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want "in the pyproject.toml"

Copy link
Member

@zanieb zanieb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is ready. It's hard to keep track of everything with the scope of the changes.

I have a few stylistic nits remaining and want to see more user-facing documentation explaining how --group interacts with other options and how it differs from the rest of the uv interface. We can push that into a separate pull request, but it's very important. I'm happy to help with that if you need it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants