Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI/Github Action Failed to extract archive: /home/runner/.cache/uv/.tmppavima/extracted #2235

Closed
anilbey opened this issue Mar 6, 2024 · 38 comments · Fixed by #11359
Closed

CI/Github Action Failed to extract archive: /home/runner/.cache/uv/.tmppavima/extracted #2235

anilbey opened this issue Mar 6, 2024 · 38 comments · Fixed by #11359
Assignees
@charliermarsh
Labels
bug Something isn't working

Comments

@anilbey
Copy link

anilbey commented Mar 6, 2024

Hello,

There is this small error that occurs only in the first CI run. When the CI is run the first time it says failed to extract archive (pointing to the .cache directory).

  Caused by: Failed to build: bluecellulab @ file:///home/runner/work/BlueCelluLab/BlueCelluLab/.tox/.tmp/package/1/bluecellulab-0.0.1.dev1.tar.gz
  Caused by: Failed to extract archive: /home/runner/.cache/uv/.tmppavima/extracted
  Caused by: failed to unpack `/home/runner/.cache/uv/.tmppavima/extracted/bluecellulab-0.0.1.dev1/tests/examples/circuit_sonata_quick_scx_multi_circuit/components/CircuitB/mo`
  Caused by: failed to unpack `bluecellulab-0.0.1.dev1/tests/examples/circuit_sonata_quick_scx_multi_circuit/components/CircuitB/mo` into `/home/runner/.cache/uv/.tmppavima/extracted/bluecellulab-0.0.1.dev1/tests/examples/circuit_sonata_quick_scx_multi_circuit/components/CircuitB/mo`
  Caused by: Is a directory (os error 21)

When I restart the CI in the 2nd attempt, it works perfectly fine. This pattern repeats in every CI run. First one fails, the second one succeeds.

here are the CI logs. The uv version, Python version and the other dependencies are all listed in the logs below.
https://github.com/BlueBrain/BlueCelluLab/actions/runs/8169106907/job/22332500653
@anilbey
Copy link
Author

anilbey commented Mar 6, 2024

Logs from the successful run (2nd attempt): https://github.com/BlueBrain/BlueCelluLab/actions/runs/8169106907/job/22334002556

@charliermarsh charliermarsh added the bug Something isn't working label Mar 6, 2024
@charliermarsh
Copy link
Member

Thanks!

@charliermarsh
Copy link
Member

My read of the trace is that for some reason bluecellulab-0.0.1.dev1/tests/examples/circuit_sonata_quick_scx_multi_circuit/components/CircuitB/mo is not marked as a directory in the archive, i.e., the headers don't indicate that it's a directory. Any idea how the source distribution is being built? (Is that path a symlink, or something like that?)

@anilbey
Copy link
Author

anilbey commented Mar 7, 2024

Oh I see. Those directories contain static files required to run some tests and those are excluded from the package.

@anilbey
Copy link
Author

anilbey commented Mar 12, 2024

With the newer versions of uv and tox-uv, restarting the workflow does not change the outcome. If the cache fails, the whole workflow fails.

I rolled back to uv==0.1.14 and tox-uv==1.4.0

@anilbey anilbey mentioned this issue Mar 13, 2024
Merged
@charliermarsh
Copy link
Member

Is there any way you could extract and share the bluecellulab-0.0.1.dev1.tar.gz file from the failing run?

@charliermarsh charliermarsh added the needs-mre Needs more information for reproduction label Mar 22, 2024
@anilbey
Copy link
Author

anilbey commented Mar 25, 2024

Unfortunately I cannot access the data from the github actions. It is not listed in the Artifacts section.

@AlessandroPomponio
Copy link

I've seen the same thing happen with the latest versions of uv as well. Sometimes it takes multiple restarts for it to work.
Unfortunately I can't provide the tar archive either :(

@anilbey
Copy link
Author

anilbey commented Apr 19, 2024

A couple of times it happened in my local as well but at that time I was in the middle of some other task and later I couldn't reproduce it.
It is a Heisenbug :)

@DetachHead
Copy link

not sure if this helps at all but i've noticed this happening much more frequently since updating from uv 0.5.20 to 0.5.24. i've added a step to my CI to upload the relevant files so i'll share that here next time it happens

@DetachHead
Copy link

ok it happened first try:

   × Failed to build `/home/runner/work/basedpyright/basedpyright`
  ├─▶ failed to unpack
  │   `/home/runner/.cache/uv/sdists-v6/.tmppLstqH/basedpyright-1.26.0/packages/pyright-internal/typeshed-fallback/stubs/aws-xray-sdk/aws_xray_sdk/core`
  ├─▶ failed to unpack
  │   `basedpyright-1.26.0/packages/pyright-internal/typeshed-fallback/stubs/aws-xray-sdk/aws_xray_sdk/core`
  │   into
  │   `/home/runner/.cache/uv/sdists-v6/.tmppLstqH/basedpyright-1.26.0/packages/pyright-internal/typeshed-fallback/stubs/aws-xray-sdk/aws_xray_sdk/core`
  ╰─▶ Is a directory (os error 21)

here's the run and here's the whole uv cache directory, but i couldn't find any of those paths inside it. do they get moved or deleted after the build fails or something? let me know if you need any more info or artifacts and i'll try to provide them

@patrickhulce
Copy link

I have the same Heisenbug-like behavior here across a few different projects. In case it helps investigate, for me at least it's always on a directory with a git submodule.

@felnne
Copy link

felnne commented Feb 4, 2025

I'm also experiencing this nearly constantly. Dropping back to 0.5.19 seems to fix it though.

@charliermarsh
Copy link
Member

Interesting -- I will take a look, thanks. (Does the submodule thing apply to anyone else's projects?)

@charliermarsh
Copy link
Member

The trace suggests there's... something wrong with the tar file itself that's been built? Like, the basedpyright-1.26.0/packages/pyright-internal/typeshed-fallback/stubs/aws-xray-sdk/aws_xray_sdk/core entry isn't being considered a "directory".

@DetachHead
Copy link

mines not a submodule. the only thing of note i can think of is that im bundling all the typeshed stubs so theres a lot of files in there

@charliermarsh
Copy link
Member

Could be related? I'm not sure. I'm gonna go through tar-rs and see if we're missing any changes from upstream in our async-oriented fork.

@felnne
Copy link

felnne commented Feb 5, 2025

Does the submodule thing apply to anyone else's projects?

Mine isn't either these are the sort of errors I get:

Building source distribution...
  × Failed to build `/builds/MAGIC/assets-tracking-service`
  ├─▶ failed to unpack
  │   `/builds/MAGIC/assets-tracking-service/.uv-cache/sdists-v7/.tmpl3VH14/assets_tracking_service-0.0.0.post45/.uv-cache/sdists-v7/pypi/arcgis/2.4.0/YNtoI0PDe5rTJdNyzq2A4/src`
  ├─▶ failed to unpack
  │   `assets_tracking_service-0.0.0.post45/.uv-cache/sdists-v7/pypi/arcgis/2.4.0/YNtoI0PDe5rTJdNyzq2A4/src`
  │   into
  │   `/builds/MAGIC/assets-tracking-service/.uv-cache/sdists-v7/.tmpl3VH14/assets_tracking_service-0.0.0.post45/.uv-cache/sdists-v7/pypi/arcgis/2.4.0/YNtoI0PDe5rTJdNyzq2A4/src`
  ╰─▶ Is a directory (os error 21)

Most recently I was getting this error consistently which prompted me to rollback to 0.5.19 (I'm not sure if it's exactly the same problem but I assume related):

$ uv build
Building source distribution...
  × Failed to build `/builds/MAGIC/assets-tracking-service`
  ├─▶ failed to unpack
  │   `/builds/MAGIC/assets-tracking-service/.uv-cache/sdists-v7/.tmpRCOa4v/assets_tracking_service-0.4.2/.uv-cache/builds-v0/.tmprScxgk/lib/python3.11/site-packages/hatchling/__about__.py`
  ╰─▶ No such file or directory (os error 2) while canonicalizing
      /builds/MAGIC/assets-tracking-service/.uv-cache/sdists-v7/.tmpRCOa4v/assets_tracking_service-0.4.2/.uv-cache/builds-v0/.tmprScxgk/lib/python3.11/site-packages/hatchling/assets_tracking_service-0.4.2/.uv-cache/archive-v0/gq0qJxV0iQijn3Zm5EYb6/hatchling/__about__.py

@charliermarsh
Copy link
Member

The second trace you referenced there is unrelated. It was fixed in v0.5.28.

@charliermarsh
Copy link
Member

I'm trying to reproduce this: #11249

@charliermarsh charliermarsh self-assigned this Feb 5, 2025
@charliermarsh
Copy link
Member

@DetachHead -- I've been trying to get this to trigger here, with a modified build that persists everything in the cache. It hasn't failed yet, though. Any ideas on where I could be divering?

@DetachHead
Copy link

only thing i can think of is the uv version is newer (i'm currently on 0.5.24), i can try bumping uv and see if it still happens. my ci is quite a mess so i'll try to see if i can make a cut down version that reproduces the issue.

@charliermarsh
Copy link
Member

I'll try bumping down to 0.5.24 for good measure.

@DetachHead
Copy link

confirmed still happening on uv 0.5.29. i made a branch that runs the same thing in a matrix of 10 jobs and removed the steps that make it take half an hour to run https://github.com/DetachHead/basedpyright/actions/runs/13215596833/job/36894495570?pr=1047

@charliermarsh
Copy link
Member

Great, thank you! I’ll try to patch in my changes and re-run.

@charliermarsh
Copy link
Member

Were you able to trigger this on your first try? I can't get it to fail.

@DetachHead
Copy link

that time it failed first try, but i ran it again with fail-fast: false in a matrix of 20 jobs and it looks like it has about a 1 in 10 chance of failing https://github.com/DetachHead/basedpyright/actions/runs/13221110183/job/36906253949?pr=1047

@charliermarsh
Copy link
Member

Ok I think I was able to reproduce and now have my hands on the failing .tar.gz file...

@charliermarsh
Copy link
Member

Ok, I've confirmed that it's this issue: astral-sh/tokio-tar#1

@charliermarsh charliermarsh removed the needs-mre Needs more information for reproduction label Feb 9, 2025
@charliermarsh
Copy link
Member

Attaching the archive that tokio-tar fails to unpack (but tar-rs unpacks without issue): basedpyright-1.26.0.tar.gz

@charliermarsh charliermarsh mentioned this issue Feb 9, 2025
Merged
@charliermarsh
Copy link
Member

I think I have a fix here: astral-sh/tokio-tar#40. It now successfully unpacks the linked archive.

@charliermarsh charliermarsh mentioned this issue Feb 9, 2025
Merged
@charliermarsh
Copy link
Member

Should be fixed in the next release.

@DetachHead
Copy link

thank you! i really appreciate your work narrowing down these difficult to reproduce issues. on any other project bugs like these would never get fixed

@charliermarsh
Copy link
Member

Thanks for your help! I'm so happy that we were able to fix it.

@DetachHead DetachHead mentioned this issue Feb 10, 2025
Merged
@patrickhulce
Copy link

i really appreciate your work narrowing down these difficult to reproduce issues. on any other project bugs like these would never get fixed

Big +1 here. This is what sets uv apart. When colleagues express adoption hesitation about uv being new, this is what I tell them and interactions like this are what make me an evangelist.

Thanks for all your work! 🙏

@thiago-pythonic
Copy link

Thanks for the quick action, @charliermarsh!

Is there a workaround to prevent this from happening while the new release isn't available? This continues to happen even after rolling back to uv 0.5.13.

@charliermarsh
Copy link
Member

No, but we'll probably release today.

@thiago-pythonic
Copy link

Sounds great! Thanks!

loic-lescoat pushed a commit to loic-lescoat/uv that referenced this issue Mar 2, 2025
@charliermarsh
Upgrade astral-tokio-tar to v0.5.1 (astral-sh#11359)
08b5f12 
## Summary

Pulling in astral-sh/tokio-tar#40.

Closes astral-sh#2235.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@charliermarsh charliermarsh

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Upgrade astral-tokio-tar to v0.5.1 astral-sh/uv
7 participants
@charliermarsh @patrickhulce @anilbey @felnne @AlessandroPomponio @DetachHead @thiago-pythonic