Use files instead of junctions on Windows #11269

charliermarsh · 2025-02-06T01:03:40Z

Summary

Instead of using junctions, we can just write files that contain (as the file contents) the target path. This requires a little more finesse in that, as readers, we need to know where to expect these. But it also means we get to avoid junctions, which have led to a variety of confusing behaviors. Further, replace_symlink should now be on atomic on Windows.

Closes #11263.

charliermarsh · 2025-02-06T02:20:02Z

We might be able to remove some of the extra flock usages we have on Windows after this, but I want to test and evaluate that separately. The proximate motivation here is that Ofek / Datadog are seeing a really strange behavior in their Windows container whereby we had a junction that... didn't point to anything? Like, it's not that the target was gone, the target was empty.

konstin · 2025-02-06T10:14:16Z

crates/uv-fs/src/lib.rs

-            ),
-        ));
-    }
+    // First, attempt to create a file at the location, but fail if it doesn't exist.


Does this intend to say "fail if it does exist"?

konstin · 2025-02-06T10:15:14Z

crates/uv-fs/src/lib.rs

+            let temp_dir = tempfile::tempdir_in(dst.as_ref().parent().unwrap())?;
+            let temp_file = temp_dir.path().join("link");


Do we need a tempdir or can we directly create a tempfile with a random name in the directory?

BurntSushi

Rolling your own symlinks because Windows-native symlinks are so bad... It's just crazy enough to work! Haha.

I made a suggestion about how to avoid the lossy call here. Sorry about being unclear last night.

I also wonder if perhaps we can emit some more logs in places, but I defer to you on that. I don't have a ton of context on this area of the code.

BurntSushi · 2025-02-06T13:11:20Z

crates/uv-cache/src/lib.rs

+                            .path()
+                            .file_name()
+                            .and_then(|file_name| file_name.to_str())
+                        else {


Should this be an error? Or perhaps a WARN log or something?

BurntSushi · 2025-02-06T13:11:31Z

crates/uv-cache/src/lib.rs

+                        else {
+                            continue;
+                        };
+                        if WheelFilename::from_stem(filename).is_err() {


BurntSushi · 2025-02-06T13:12:01Z

crates/uv-cache/src/lib.rs

+                        if WheelFilename::from_stem(filename).is_err() {
+                            continue;
+                        }
+                        if let Ok(target) = uv_fs::resolve_symlink(entry.path()) {


And here.

Although I guess we weren't emitting logs before. I wonder if it might help debugging if something goes wrong here.

BurntSushi · 2025-02-06T13:20:31Z

crates/uv-fs/src/lib.rs

+        Ok(mut file) => {
+            // Write the target path to the file.
+            use std::io::Write;
+            file.write_all(src.as_ref().to_string_lossy().as_bytes())?;


Oh, okay, now that I know what you are doing here, I think I can suggest something more bullet proof that isn't too much work. Basically, on Windows, file paths are just arbitrary sequences of u16 (with some restrictions), just like on Unix, file paths are just arbitrary sequences of u8. So the way to avoid the lossy conversion here is to just store the u16 sequence. You can get the raw u16 sequence from this API in std: https://doc.rust-lang.org/std/os/windows/ffi/trait.OsStrExt.html

Once you have a Vec<u16> build in memory, create a Vec<u8> with twice the length and then use byteorder to (safely) write the &[u16] into a &mut [u8]: https://docs.rs/byteorder/latest/byteorder/trait.ByteOrder.html#tymethod.write_u16_into

Then save that to the file.

To go in the other direction, use ByteOrder::read_u16_into and std::os::windows::ffi::OsStringExt.

When using byteorder, I'd suggest picking a specific byteorder (byteorder::BigEndian) and always using it. Regardless of the endianness of the host platform. That way, the file format is portable.

Awesome, thank you!

charliermarsh · 2025-02-07T02:35:06Z

Ok, I simplified things a bit (I think). Instead of "generically" using these routines to read and write symlinks, they're now methods on the cache specifically framed at writing links to archive entries and reading links to archive entries.

charliermarsh · 2025-02-07T02:37:19Z

(I didn't mean to close, accidental click.)

ofek · 2025-02-07T02:39:24Z

In the future, would it be possible to detect symlink capability on Windows and use this new approach as a fallback?

## Summary Instead of using junctions, we can just write files that contain (as the file contents) the target path. This requires a little more finesse in that, as readers, we need to know where to expect these. But it also means we get to avoid junctions, which have led to a variety of confusing behaviors. Further, `replace_symlink` should now be on atomic on Windows. Closes #11263.

Instead of using junctions, we can just write files that contain (as the file contents) the target path. This requires a little more finesse in that, as readers, we need to know where to expect these. But it also means we get to avoid junctions, which have led to a variety of confusing behaviors. Further, `replace_symlink` should now be on atomic on Windows. Closes #11263.

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | minor | `0.5.31` -> `0.6.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.6.0`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#060) [Compare Source](astral-sh/uv@0.5.31...0.6.0) There have been 31 releases and 1135 pull requests since [0.5.0](https://github.com/astral-sh/uv/releases/tag/0.5.0), our last release with breaking changes. As before, we've accumulated various changes that improve correctness and user experience, but could break some workflows. This release contains those changes; many have been marked as breaking out of an abundance of caution. We expect most users to be able to upgrade without making changes. ##### Breaking changes - **Create `main.py` instead of `hello.py` in `uv init`** ([#10369](astral-sh/uv#10369)) Previously, `uv init` created a `hello.py` sample file. Now, `uv init` will create `main.py` instead — which aligns with expectations from user feedback. The `--bare` option can be used to avoid creating the file altogether. - **Respect `UV_PYTHON` in `uv python install`** ([#11487](astral-sh/uv#11487)) Previously, `uv python install` did not read this environment variable; now it does. We believe this matches user expectations, however, this will take priority over `.python-version` files which could be considered breaking. - **Set `UV` to the uv executable path** ([#11326](astral-sh/uv#11326)) When uv spawns a subprocess, it will now have the `UV` environment variable set to the `uv` binary path. This change is breaking if you are setting the `UV` environment variable yourself, as we will overwrite its value. Additionally, this change requires marking the uv Rust entrypoint (`uv::main`) as `unsafe` to avoid unsoundness — this is only relevant if you are invoking uv using Rust. See the [Rust documentation](https://doc.rust-lang.org/std/env/fn.set_var.html#safety) for details about the safety of updating a process' environment. - **Error on non-existent extras, e.g., in `uv sync`** ([#11426](astral-sh/uv#11426)) Previously, uv would silently ignore non-existent extras requested on the command-line (e.g., via `uv sync --extra foo`). This is *generally* correct behavior when resolving requests for package extras, because an extra may be present on one compatible version of a package but not another. However, this flexibility doesn't need to apply to the local project and it's less surprising to error here. - **Error on missing dependency groups when `--frozen` is provided** ([#11499](astral-sh/uv#11499)) Previously, uv would not validate that the requested dependency groups were present in the lockfile when the `--frozen` flag was used. Now, an error will be raised if a requested dependency group is not present. - **Change `-p` to a `--python` alias in `uv pip compile`** ([#11486](astral-sh/uv#11486)) In `uv pip compile`, `-p` was an alias for `--python-version` while everywhere else in uv's interface it is an alias for `--python`. Additionally, `uv pip compile` did not respect the `UV_PYTHON` environment variable. Now, the semantics of this flag have been updated for parity with the rest of the CLI. However, `--python-version` is unique: if we cannot find an interpreter with the given version, we will not fail. Instead, we'll use an alternative interpreter and override its version tags with the requested version during package resolution. This behavior is retained here for backwards compatibility, `--python <version>` / `-p <version>` will not fail if the version cannot be found. However, if a specific interpreter is requested, e.g., with `--python <path>` or `--python pypy`, and cannot be found — uv will exit with an error. The breaking changes here are that `UV_PYTHON` is respected and `--python <version>` will no longer fail if the version cannot be found. - **Bump `alpine` default tag to 3.21 for derived Docker images** ([#11157](astral-sh/uv#11157)) Alpine 3.21 was released in Dec 2024 and is used in the official Alpine-based Python images. Our `uv:python3.x-alpine` images have been using 3.21 since uv v0.5.8. However, now the the `uv:alpine` image will use 3.21 instead of 3.20 and `uv:alpine3.20` will no longer be updated. - **Use files instead of junctions on Windows** ([#11269](astral-sh/uv#11269)) Previously, we used junctions for atomic replacement of cache entries on Windows. Now, we use a file with a pointer to the cache entry instead. This resolves various edge-case behaviors with junctions. These files are only intended to be consumed by uv and the cache version has been bumped. We do not think this change will affect workflows. ##### Stabilizations - **`uv publish` is no longer in preview** ([#11032](astral-sh/uv#11032)) This does not come with any behavior changes. You will no longer see an experimental warning when using `uv publish`. See the linked pull request for a report on the stabilization. ##### Enhancements - Support `--active` for PEP 723 script environments ([#11433](astral-sh/uv#11433)) - Add `revision` to the lockfile to allow backwards-compatible metadata changes ([#11500](astral-sh/uv#11500)) ##### Bug fixes - Avoid reading metadata from `.egg-info` files ([#11395](astral-sh/uv#11395)) - Include archive bucket version in archive pointers ([#11306](astral-sh/uv#11306)) - Omit lockfile version when additional fields are dynamic ([#11468](astral-sh/uv#11468)) - Respect executable name in `uvx --from tool@latest` ([#11465](astral-sh/uv#11465)) ##### Documentation - The `CHANGELOG.md` is now split into separate files for each "major" version to fix rendering ([#11510](astral-sh/uv#11510)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

Instead of using junctions, we can just write files that contain (as the file contents) the target path. This requires a little more finesse in that, as readers, we need to know where to expect these. But it also means we get to avoid junctions, which have led to a variety of confusing behaviors. Further, `replace_symlink` should now be on atomic on Windows. Closes astral-sh#11263.

charliermarsh force-pushed the charlie/sym branch from 52b556d to cfc11eb Compare February 6, 2025 01:04

charliermarsh added the no-build Disable building binaries in CI label Feb 6, 2025

charliermarsh force-pushed the charlie/sym branch 7 times, most recently from a4bc912 to 4e21181 Compare February 6, 2025 01:54

charliermarsh requested review from BurntSushi and konstin February 6, 2025 02:01

charliermarsh added the windows Specific to the Windows platform label Feb 6, 2025

charliermarsh marked this pull request as ready for review February 6, 2025 02:02

charliermarsh force-pushed the charlie/sym branch 2 times, most recently from 7a01a26 to 4a3c71c Compare February 6, 2025 02:08

charliermarsh requested a review from zanieb February 6, 2025 02:18

konstin reviewed Feb 6, 2025

View reviewed changes

konstin approved these changes Feb 6, 2025

View reviewed changes

BurntSushi reviewed Feb 6, 2025

View reviewed changes

charliermarsh requested a review from Gankra February 6, 2025 17:14

charliermarsh force-pushed the charlie/sym branch from 4a3c71c to 42af524 Compare February 7, 2025 02:34

charliermarsh changed the base branch from main to charlie/version February 7, 2025 02:34

charliermarsh closed this Feb 7, 2025

charliermarsh reopened this Feb 7, 2025

charliermarsh added this to the v0.6.0 milestone Feb 7, 2025

zanieb mentioned this pull request Feb 7, 2025

v0.6.0 #11329

Merged

charliermarsh force-pushed the charlie/version branch from 8283542 to 831df6c Compare February 7, 2025 22:07

Base automatically changed from charlie/version to tracking/060 February 7, 2025 22:20

charliermarsh added 3 commits February 7, 2025 17:24

Use files instead of junctions on Windows

2d20c8a

Make it relative

7a554ff

Use a link abstraction instead of writing a path

f52f9e4

charliermarsh force-pushed the charlie/sym branch 3 times, most recently from ccc1326 to 439e70f Compare February 7, 2025 23:12

Restore junctions for toolchain migration

0a2f662

charliermarsh force-pushed the charlie/sym branch from 439e70f to 0a2f662 Compare February 7, 2025 23:35

charliermarsh merged commit 238bc29 into tracking/060 Feb 8, 2025
83 checks passed

charliermarsh deleted the charlie/sym branch February 8, 2025 00:13

BrewTestBot mentioned this pull request Feb 14, 2025

uv 0.6.0 Homebrew/homebrew-core#207738

Merged

marcelotduarte mentioned this pull request Feb 19, 2025

uv's tool installation feature breaks on Wine #5090

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use files instead of junctions on Windows #11269

Use files instead of junctions on Windows #11269

charliermarsh commented Feb 6, 2025 •

edited

Loading

charliermarsh commented Feb 6, 2025

konstin Feb 6, 2025 •

edited

Loading

charliermarsh Feb 6, 2025

konstin Feb 6, 2025

BurntSushi left a comment

BurntSushi Feb 6, 2025

BurntSushi Feb 6, 2025

BurntSushi Feb 6, 2025

BurntSushi Feb 6, 2025

charliermarsh Feb 6, 2025

charliermarsh commented Feb 7, 2025

charliermarsh commented Feb 7, 2025

ofek commented Feb 7, 2025

		let temp_dir = tempfile::tempdir_in(dst.as_ref().parent().unwrap())?;
		let temp_file = temp_dir.path().join("link");

Use files instead of junctions on Windows #11269

Use files instead of junctions on Windows #11269

Conversation

charliermarsh commented Feb 6, 2025 • edited Loading

Summary

charliermarsh commented Feb 6, 2025

konstin Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

charliermarsh Feb 6, 2025

Choose a reason for hiding this comment

konstin Feb 6, 2025

Choose a reason for hiding this comment

BurntSushi left a comment

Choose a reason for hiding this comment

BurntSushi Feb 6, 2025

Choose a reason for hiding this comment

BurntSushi Feb 6, 2025

Choose a reason for hiding this comment

BurntSushi Feb 6, 2025

Choose a reason for hiding this comment

BurntSushi Feb 6, 2025

Choose a reason for hiding this comment

charliermarsh Feb 6, 2025

Choose a reason for hiding this comment

charliermarsh commented Feb 7, 2025

charliermarsh commented Feb 7, 2025

ofek commented Feb 7, 2025

charliermarsh commented Feb 6, 2025 •

edited

Loading

konstin Feb 6, 2025 •

edited

Loading