-
-
Notifications
You must be signed in to change notification settings - Fork 31.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_unicode
always fails during a PGO-optimised build (because it does not exist)
#110276
Comments
…ld (#110277) `test_unicode` does not exist
I don't understand this issue. We have many PGO buildbots. How is it possible that a test was removed and nothing broke? If I replace PGO_TESTS with PGO_TESTS=['test_xxx'], tests fail as expected:
Maybe the problem is that Makefile.pre.in ignores failures :-(
@corona10: Any idea why we ignore failures? |
FWIW, the failure wasn't "just on my machine": you can see the "test_unicode failed (uncaught exception)" message in the logs for our PGO buildbots too. Example: https://buildbot.python.org/all/#/builders/249/builds/6518/steps/3/logs/stdio.
Hmm, I'm on Windows, and the test failure doesn't cause a PGO build (which doesn't use the Makefile on Windows) to fail on my machine either. The test fails as part of the build, the failure is reported and logged to the terminal, but the build then continues and is marked as a success. |
A log is not really what I expect when I say "failure". I expected a red buildbot :-) Otherwise, it's too easy to miss. |
Oh, we agree! There's definitely a problem here that this issue went unseen for so long! :D This is the reason why I haven't closed the issue yet, even though #110277 has now been merged, fixing the immediate problem. |
I wrote PR #110295 to no longer ignore PROFILE_TASK failure silently. |
My commit 6ab6040 no longer ignores PROFILE_TASK failure silently. |
PGO builds are tested daily on Unix by buildbots. So now we should get any similar issue get reported way earlier. |
…on#110295) (cherry picked from commit 6ab6040)
…GO build (python#110277) `test_unicode` does not exist
Bug report
Bug description:
As part of a PGO-optimised build, a selection of tests are run in order for the PGO build to take place (if I understand correctly, these tests are used to gather data on which functions in Python are performance-critical and should be optimised). Exactly which tests are run is hardcoded here:
cpython/Lib/test/libregrtest/pgo.py
Lines 1 to 51 in bb2e96f
test_unicode
is specified as one of the tests to be run as part of this selection. However,test_unicode
always fails during a PGO build. Why? Because it was renamed astest_str
in #13172 (by @asqui). This hasn't been noticed until now because the PGO build is still marked as a "success" even if some tests fail during the analysis phase.The fix for the immediate issue is simple: don't try to run
test_unicode
as part of the PGO-optimised build (it doesn't exist anymore); runtest_str
instead. Longer term, though, we might want to look at making the PGO build fail if it tries to run nonexistent tests. It's probably correct for the build to continue even if some tests fail during the analysis phase, but perhaps we could make an exception for this specific kind of failure (attempting to run nonexistent tests).Cc. @hugovk as the primary reviewer for #13172, and @vstinner as a libregrtest expert.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Windows
Linked PRs
test_str
, nottest_unicode
, as part of the PGO build #110277The text was updated successfully, but these errors were encountered: