pip fails to resolve multiple dependencies #11352

mluypaert · 2025-01-20T14:22:03Z

Is there an existing issue for this?

I have searched the existing issues

Package ecosystem

pip

Package manager version

pip-compile 7.4.1

Language version

Python 3.12(.3)

Manifest location and content before the Dependabot update

https://github.com/alliance-genome/agr_pavi/blob/856e6bb5933c00ad475db0023746445a66b24b4f/api/tests/requirements.txt
https://github.com/alliance-genome/agr_pavi/blob/main/api/pyproject.toml

dependabot.yml content

https://github.com/alliance-genome/agr_pavi/blob/856e6bb5933c00ad475db0023746445a66b24b4f/.github/dependabot.yml

Updated dependency

mypy from 1.13.0 to 1.14.1
smart-open[s3] from 7.0.5 to 7.1.0

What you expected to see, versus what you actually saw

Dependabot should be able to propose a PR to update mypy from 1.13(.0) to 1.14(.1), and to update smart-open from 7.0(.5) to 7.1(.0). Dependabot config has not been updated and dependabot used to be able to propose updates for both packages in the past.

Native package manager behavior

Successful updates.

Images of the diff or a link to the PR, issue, or logs

Likely related to changes introduced in merged dependabot PR https://github.com/dependabot/dependabot-core/pull/11305/files#diff-e97dac51f6d0eb39115b1c017916def375cd4cfe8743bea989f9b1d8e04ff791, released as dependabot v0.293.0.

Dependabot run mypy update error:

updater | 2025/01/20 09:23:45 INFO <job_950382875> Updating mypy from 1.13.0 to 1.14.1
updater | 2025/01/20 09:23:45 INFO <job_950382875> Started process PID: 1940 with command: pyenv exec python3 /opt/python/run.py
  proxy | 2025/01/20 09:23:47 [075] WARN: Cannot handshake client pypi.org:443 local error: tls: bad record MAC
updater | 2025/01/20 09:23:47 INFO <job_950382875> Process PID: 1940 completed with status: pid 1940 exit 1
2025/01/20 09:23:47 INFO <job_950382875> Total execution time: 1.23 seconds
updater | 2025/01/20 09:23:47 WARN <job_950382875> Traceback (most recent call last):
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 1319, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              encode_chunked=req.has_header('Transfer-encoding'))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1091, in _send_output
    self.send(msg)
    ~~~~~~~~~^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1035, in send
    self.connect()
    ~~~~~~~~~~~~^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1477, in connect
    self.sock = self._context.wrap_socket(self.sock,
                ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
                                          server_hostname=server_hostname)
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1018)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run.py", line 18, in <module>
    print(hasher.get_dependency_hash(*args["args"]))
          ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/opt/python/lib/hasher.py", line 11, in get_dependency_hash
    hashes = hashin.get_package_hashes(
        dependency_name,
    ...<2 lines>...
        index_url=index_url
    )
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/site-packages/hashin.py", line 654, in get_package_hashes
    data = get_package_data(package, index_url, verbose)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/site-packages/hashin.py", line 582, in get_package_data
    content = json.loads(_download(url))
                         ~~~~~~~~~^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/site-packages/hashin.py", line 60, in _download
    r = urlopen(url)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 1367, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        context=self._context)
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 1322, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1018)>

Dependabot smart-open update error:

updater | 2025/01/20 09:23:52 INFO <job_950382875> Updating smart-open[s3] from 7.0.5 to 7.1.0
updater | 2025/01/20 09:23:52 INFO <job_950382875> Started process PID: 2076 with command: pyenv exec python3 /opt/python/run.py
  proxy | 2025/01/20 09:23:53 [201] WARN: Cannot handshake client pypi.org:443 local error: tls: bad record MAC
updater | 2025/01/20 09:23:53 INFO <job_950382875> Process PID: 2076 completed with status: pid 2076 exit 1
2025/01/20 09:23:53 INFO <job_950382875> Total execution time: 1.26 seconds
updater | 2025/01/20 09:23:53 WARN <job_950382875> Traceback (most recent call last):
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 1319, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              encode_chunked=req.has_header('Transfer-encoding'))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1091, in _send_output
    self.send(msg)
    ~~~~~~~~~^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1035, in send
    self.connect()
    ~~~~~~~~~~~~^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1477, in connect
    self.sock = self._context.wrap_socket(self.sock,
                ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
                                          server_hostname=server_hostname)
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1018)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run.py", line 18, in <module>
    print(hasher.get_dependency_hash(*args["args"]))
          ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/opt/python/lib/hasher.py", line 11, in get_dependency_hash
    hashes = hashin.get_package_hashes(
        dependency_name,
    ...<2 lines>...
        index_url=index_url
    )
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/site-packages/hashin.py", line 654, in get_package_hashes
    data = get_package_data(package, index_url, verbose)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/site-packages/hashin.py", line 582, in get_package_data
    content = json.loads(_download(url))
                         ~~~~~~~~~^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/site-packages/hashin.py", line 60, in _download
    r = urlopen(url)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 1367, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        context=self._context)
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.13.1/lib/python3.13/urllib/request.py", line 1322, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1018)>
  proxy | 2025/01/20 09:23:53 [203] POST /update_jobs/950382875/record_update_job_error
2025/01/20 09:23:53 [203] 204 /update_jobs/950382875/record_update_job_error
updater | 2025/01/20 09:23:53 INFO <job_950382875> Handled error whilst updating smart-open[s3]: dependency_file_not_resolvable {:message=>"Error resolving dependency."}
  proxy | 2025/01/20 09:23:53 [205] POST /update_jobs/950382875/record_ecosystem_meta
2025/01/20 09:23:53 [205] 204 /update_jobs/950382875/record_ecosystem_meta

Smallest manifest that reproduces the issue

https://github.com/alliance-genome/agr_pavi/blob/856e6bb5933c00ad475db0023746445a66b24b4f/api/aws_infra/tests/requirements.txt

The text was updated successfully, but these errors were encountered:

hartwork · 2025-01-21T18:22:55Z

I think it's worth repeating error "Error resolving dependency." verbatim here, maybe even worth putting that into the title.

An what's particularly important to mention is that this issue blocks (or delays) Django security updates, at least the part where Dependabot would normally help out with a pull request.

Here's how users see the issue (if helpful):

Would be great to have that fixed. Thanks for all the time Dependabot was working well, it's an invaluable tool 🙏

PS: The issue affects (at least) two repositories of mine:

GitHub Repository: https://github.com/hartwork/jawanndenn/

Version Update Job: https://github.com/hartwork/jawanndenn/network/updates/951421641
Actions Run: https://github.com/hartwork/jawanndenn/actions/runs/12892921378/job/35948222890
Log (attached): dependabot-jawanndenn-log.txt

GitHub Repository: https://github.com/hartwork/wnpp.debian.net/

Version Update Job: https://github.com/hartwork/wnpp.debian.net/network/updates/951421763
Actions Run: https://github.com/hartwork/wnpp.debian.net/actions/runs/12892926582/job/35948241563
Log (attached): dependabot-wnpp-debian-net-log.txt

hartwork · 2025-01-21T18:40:14Z

PS: I'm also just noticing that Dependabot was already failing for two weeks without notifying anyone:

(from https://github.com/hartwork/wnpp.debian.net/network/updates/8399467/jobs)

.python-version file is used by gh-actions and dependabot. Attempt to circumvent dependabot/dependabot-core#11352

circumvents dependabot/dependabot-core#11352

mluypaert · 2025-01-28T12:33:23Z

I tested this again with the latest dependabot release (v0.294.0), and the issue persists.

However, I found a workaround to make it work again for me:
I noticed in the logs that dependabot was using python 3.13, while my project is written for python 3.12. While dependabot is supposed to detect python 3.12 from the requirements.txt files generated through pip-compile (according to #4216 and #8280), it doesn't seem to do this and went for 3.13 instead.
To fix this behavior, I added a .python-version file to every python component directory in my repository, with 3.12 in it to force dependabot to use python 3.12, and once merged dependabot functioned correctly.

Two takeaways from this though:

Time to put work in documenting this python version selection mechanism, as it took me a lot of digging through various issue tickets to figure out I could use .python-version for this. This now seems to be a requirement to make dependabot work correctly rather than just an implementation detail (@jeffwidman). Document how the Python version is determined and more #10291
Please implement support to detect and use the requires-python version specified in the pyproject.toml file, which is PEP-621 standard. This would avoid needing to specify duplicate versions in .python-version files Python version from pyproject.toml requires-python (PEP621) #8244

.. because of defunct Dependabot: dependabot/dependabot-core#11352

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork · 2025-01-28T14:04:39Z

If of use to anyone, here's how I mass-bumped non-pip-tools requirements with hashes today using hashin while Dependabot is on sick leave:

for i in requirements*.txt ; do grep == "$i" | awk -F= '{print $1}' | xargs -r -t -n1 hashin -r "$i" ; done

hartwork · 2025-02-14T18:46:19Z

@dependabot could you comment about an ETA for a fix? This issue renders Dependabot dysfunctional for at least two of my repositories — this issue matters. Could you raise priority please?

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork · 2025-02-24T17:11:46Z

@crittermike can you help get this showstopper issue priority and a reply?

.. rather than Python 3.13 to try trick GitHub Dependabot back into operation for a workaround. dependabot/dependabot-core#11352

abdulapopoola · 2025-03-10T17:02:29Z

Hi @hartwork , sorry for the late response. Thanks for bringing this to our notice

mluypaert added the T: bug 🐞 Something isn't working label Jan 20, 2025

github-project-automation bot added this to Dependabot Jan 20, 2025

github-actions bot added the L: python label Jan 20, 2025

hartwork mentioned this issue Jan 21, 2025

pip failing with SSL error when updating websockets on python 3.13 #11323

Open

1 task

mluypaert added a commit to alliance-genome/agr_pavi that referenced this issue Jan 27, 2025

Simplified python-version management.

08ffd39

.python-version file is used by gh-actions and dependabot. Attempt to circumvent dependabot/dependabot-core#11352

mluypaert mentioned this issue Jan 27, 2025

Simplified python-version management. alliance-genome/agr_pavi#486

Merged

mluypaert added a commit to alliance-genome/agr_pavi that referenced this issue Jan 28, 2025

Specified python version to be used by dependabot.

b399e0d

circumvents dependabot/dependabot-core#11352

hartwork added a commit to hartwork/wnpp.debian.net that referenced this issue Jan 28, 2025

Mass-update requirements manually

c7ecc20

.. because of defunct Dependabot: dependabot/dependabot-core#11352

hartwork added a commit to hartwork/wnpp.debian.net that referenced this issue Jan 28, 2025

Mass-update requirements manually

6fdde3a

.. because of defunct Dependabot: dependabot/dependabot-core#11352

hartwork added a commit to hartwork/wnpp.debian.net that referenced this issue Jan 28, 2025

requirements: Mass-bump manually

9053702

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork added a commit to hartwork/jawanndenn that referenced this issue Jan 28, 2025

requirements: Mass-bump manually

1f10553

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork added a commit to hartwork/jawanndenn that referenced this issue Feb 14, 2025

requirements: Mass-bump manually

c79c0f5

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork mentioned this issue Feb 14, 2025

requirements: Mass-bump manually hartwork/jawanndenn#1313

Merged

hartwork added a commit to hartwork/wnpp.debian.net that referenced this issue Feb 14, 2025

requirements: Mass-bump manually

24184e1

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork mentioned this issue Feb 14, 2025

requirements: Mass-bump manually hartwork/wnpp.debian.net#1246

Merged

matmair mentioned this issue Feb 14, 2025

Fix dependabot version detection inventree/InvenTree#9080

Merged

hartwork added a commit to hartwork/jawanndenn that referenced this issue Feb 20, 2025

requirements: Mass-bump manually

90aead7

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork mentioned this issue Feb 20, 2025

requirements: Mass-bump manually hartwork/jawanndenn#1314

Merged

hartwork added a commit to hartwork/wnpp.debian.net that referenced this issue Feb 20, 2025

requirements: Mass-bump manually

4aae70f

.. because of Dependabot's sick leave: dependabot/dependabot-core#11352

hartwork mentioned this issue Feb 20, 2025

requirements: Mass-bump manually hartwork/wnpp.debian.net#1247

Merged

hartwork mentioned this issue Feb 24, 2025

Pretend that the project is using Python 3.12 via file .python-version hartwork/wnpp.debian.net#1249

Merged

hartwork mentioned this issue Feb 24, 2025

Pretend that the project is using Python 3.12 via file .python-version hartwork/jawanndenn#1317

Merged

gegoune mentioned this issue Mar 6, 2025

pip no longer updates requirement files along with pyproject #11750

Open

1 task

mreid-tt mentioned this issue Mar 15, 2025

Bump jinja2 from 3.1.5 to 3.1.6 SynoCommunity/spkrepo#146

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip fails to resolve multiple dependencies #11352

pip fails to resolve multiple dependencies #11352

mluypaert commented Jan 20, 2025

hartwork commented Jan 21, 2025

hartwork commented Jan 21, 2025

mluypaert commented Jan 28, 2025 •

edited

Loading

hartwork commented Jan 28, 2025

hartwork commented Feb 14, 2025 •

edited

Loading

hartwork commented Feb 24, 2025

abdulapopoola commented Mar 10, 2025

pip fails to resolve multiple dependencies #11352

pip fails to resolve multiple dependencies #11352

Comments

mluypaert commented Jan 20, 2025

Is there an existing issue for this?

Package ecosystem

Package manager version

Language version

Manifest location and content before the Dependabot update

dependabot.yml content

Updated dependency

What you expected to see, versus what you actually saw

Native package manager behavior

Images of the diff or a link to the PR, issue, or logs

Smallest manifest that reproduces the issue

hartwork commented Jan 21, 2025

hartwork commented Jan 21, 2025

mluypaert commented Jan 28, 2025 • edited Loading

hartwork commented Jan 28, 2025

hartwork commented Feb 14, 2025 • edited Loading

hartwork commented Feb 24, 2025

abdulapopoola commented Mar 10, 2025

mluypaert commented Jan 28, 2025 •

edited

Loading

hartwork commented Feb 14, 2025 •

edited

Loading