Transport & pytest: garbage cannot be removed [Errno 39]

[khsrali]

To reproduce

When running pytest test_all_plugins.py -s, despite all tests passing successful, a long tail of warning messages is printed:

..
  warnings.warn(
/home/**/lib/python3.12/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-khosra_a/garbage-82a5592f-5337-4d9c-934e-7a6ace59c9a7/popen-gw0/remote7
<class 'OSError'>: [Errno 39] Directory not empty: 'remote7'
  warnings.warn(
/home/**/lib/python3.12/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-khosra_a/garbage-82a5592f-5337-4d9c-934e-7a6ace59c9a7/popen-gw0
<class 'OSError'>: [Errno 39] Directory not empty: 'popen-gw0'
  warnings.warn(
/home/**/lib/python3.12/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-khosra_a/garbage-82a5592f-5337-4d9c-934e-7a6ace59c9a7/popen-gw4/remote8
<class 'OSError'>: [Errno 39] Directory not empty: 'remote8'
..

that pytest cannot clean the garbage.

To be investigated. Also pinging @unkcpz

[FARDEEN-785]

Hey if Its not deleting Why not delete it Manually and then runs the Test

[khsrali]

Because we don't want to do it manually, and because it should not happen. :-)

[FARDEEN-785]

Because we don't want to do it manually, and because it should not happen. :-)

Can't we can use Force method to delete

[unkcpz]

The issue me relate to pytest-dev/pytest#12381, which may or may not resolved from our side. If someone what to pick this issue, would be nice to have a deep understanding on problem of data racing introduced by pytest-xdist and how to avoid it nicely.

[FARDEEN-785]

Well I am new to this and don’t have deep understanding but still I will try to resolve this issue

[crs7617]

hello @khsrali , is this issue still open? If yes, can I proceed with this ?

[khsrali]

Hi @crs7617,

I think also @ayushjariyal showed some interest working on this.
I don't know if he's still working on it or not.

[ayushjariyal]

@khsrali I am working on it. Right now, my semester exams are going on, so I am unable to give my full time to this. However, I am trying my best.

[khsrali]

It's all good, no hurry! take your time.

[crs7617]

@khsrali Thanks for the update! if there's anything I can help with, feel free to let me know.

[crs7617]

@khsrali, I noticed that @ayushjariyal is currently working on issue #6728. I just wanted to clarify if issue #6751 is separate from that, or if they are related. If they are distinct, I’d be happy to proceed with #6751. Let me know what you think!

[khsrali]

They are independent issues.
I don't know. Maybe @ayushjariyal should answer, if he's also working on that.
For me, either way is ok.

[crs7617]

alright, thanks for the clarification !

[khsrali]

@crs7617 in case you also have a solution for #6728, please feel free to provide it.
I think solving that issue is kinda high priority.

[crs7617]

@crs7617 in case you also have a solution for #6728, please feel free to provide it. I think solving that issue is kinda high priority.

sure , I had an idea about #6751 , I am just going through #6728 rn

[khsrali]

Hi @crs7617 I had a solution for #6728 in mind, so I put it on #6776 you can give us your opinion on that solution!

if you already have a solution of this current issue #6751, please don't hesitate to provide it.
Cheers!

[ayushjariyal]

Hey @khsrali ,

I investigated the issue thoroughly and found that Aiida's codebase does not have any errors related to garbage file management. The problem lies in Pyenv, where some directories are not being deleted properly because they are still not empty when Pytest attempts to remove them in pathlib.py.

To resolve this, I updated the function to forcefully delete those files,

and after the update, the warnings no longer appeared.

Could you please let me know if I missed anything or made any mistakes?

[khsrali]

Hi @ayushjariyal,
Thanks a lot for looking into this.
May you please direct me where this function, rm_rf is defined? is this something you wrote? or is it part of pyenv?
if that's the case, maybe it makes more sense to fix this in the upstream

[ayushjariyal]

Hi @khsrali ,
You're welcome! The rm_rf function is part of pyenv and can be found at:

/home/user/.pyenv/versions/3.9.20/lib/python3.9/site-packages/_pytest/pathlib.py

Since it belongs to pyenv, it might make sense to fix this upstream, but I'm not quite sure how to go about it.

[ayushjariyal]

You can find it on github here: https://github.com/pytest-dev/pytest/blob/main/src/_pytest/pathlib.py#L162

[khsrali]

Alright!

Thanks @ayushjariyal ! in that case, I think we should open an issue and ask pytest.
Can you please take care of that? Explaining the issue clearly and your proposed solution. Please also include your setup info. Both python version and pytest version.

Once the issue is opened, please also put a link here, so we can track to see what happens.

Cheers!

[ayushjariyal]

Sure! I'll open and issue with pytest.

[ayushjariyal]

I opened an issue regarding this problem. You can find it here:
pytest-dev/pytest#13269

[adityagh006]

Confirm the Issue Scope
1.Run pytest --disable-warnings to check if the issue is purely related to cleanup.
2.Verify whether the warnings affect test execution or only appear as logs.

Potential Workarounds
1.Implement a retry mechanism with a short delay before deletion to handle potential file locks.
2.Use shutil.rmtree(..., ignore_errors=True) as a temporary solution while waiting for an upstream fix.
3.Check if pytest.TempPathFactory offers a more reliable cleanup approach.

Upstream Considerations
1.Since rm_rf is part of pytest, confirm if similar issues exist in their repository.
2.If not already done, open an issue upstream detailing the behavior and environment setup.
3.Track upstream resolution and determine if a temporary fix is necessary in Aiida.

Further Investigation
1.Identify specific conditions where rm_rf fails—whether due to permissions, open file handles, or other factors.
2.Test alternative deletion methods under different OS environments to check consistency.

[khsrali]

@adityagh006, chatgpt? 🙃

It's ok to use chatgpt for coding and development.
However, the proper use of chatgpt is to actually try and develop and you mange to refine and make it work.
The plain suggestions, as you put here, we could also get ourselves 💁