setup.py install doesn't work for packages with setup_requires on older macOS

[benjaminjkraft]

SInce the TLS 1.0/1.1 turndown, easy_install doesn't work on older macOS, because the builtin libraries don't support SSL. For the most part, we just use latest pip instead, and things work fine. But when installing a package with setup_requires from source (e.g. pip install git+git://github.com/pycqa/flake8.git@master), we run into trouble even on latest pip/setuptools, because pip runs setup.py install, which in turn runs easy_install, which fails. If easy_install is completely deprecated, setuptools should call out to pip instead; or easy_install do what pip does to make TLS 1.2 work.

See also pypa/packaging-problems#134.

[jaraco]

Once pip with PEP 518 support is released, setup_requires become deprecated. So the best recommendation is for packages to declare their build dependencies in their pyproject.toml and for users to use pip 10 and install their packages that way. The pyproject.toml should ensure the build requirements are present, preventing the need for setuptools to invoke easy_install.

Also, am I right in thinking this issue only happens to very old Pythons, than 2.7.9+ supports the requisite TLS? In that case, another workaround is simply not to use old Pythons (if you need 2.7, get it from Homebrew, but really you should be using Python 3).

And if you're really in a constrained environment where you're stuck on old Pythons, consider deploying your own devpi instance to provide compatibility.

If it were straightforward to support TLS 1.2 in easy_install or to somehow rely on pip to install setup_requires, I'd be in support of either, but my suspicion is that either of those approaches are fraught with bootstrapping challenges, so it's not worth the effort in light of the alternatives above.

[benjaminjkraft]

It's happening on 2.7.10 which is system python on Sierra -- I believe it's a matter of openssl versions, not python versions. Switching to homebrew's python does fix it; I think they must ship their own openssl. (We are stuck on python 2 at present.)

But anyway, totally fair, and thanks for the response. We'll keep an eye out for pip 10 in the meantime.

[ilovezfs]

@jaraco the suggestion in pypa/packaging-problems#134 (comment) is to fall back on SecureTransport as pip does.

[pudquick]

For explicit clarification: For any installs using the system-provided python on any macOS version earlier than 10.13.0, you will run into this issue not because of old python, but because those versions are linked against OpenSSL 0.9.8 and do not support TLS 1.1 or later.

easy_install would indeed need to be patched to swap in SecureTransport usage in a similar fashion.

[pganssle]

Are we going to fix this, or should we just say that setup_requires is deprecated now that pip has PEP 518 support?

[jaraco]

It turns out PIP 518/517 support isn't sufficient, due to usage of setup.py in other situations.

I ran into this issue on the latest macOS 10.14.2 with Python 3.5.4 from python.org (the latest available binary) using tox which invokes setup.py --name (tox-dev/tox#809). Probably this needs to be fixed.

[jaraco]

In this comment, Donald suggests:

use SecureTransport like pip has

Looking at the pip codebase, here's where SecureTransport is injected, but unfortunately, it relies on urllib3, and setuptools uses stdlib for its SSL, so Setuptools would need to be updated to use urllib3 to adopt this technique.

or an option that would fallback to pyOpenSSL

Hmm...

[jaraco]

In 7fa18bf, I've drafted an attempt to use PyOpenSSL, but it fails with this traceback:

$ python3.5 -m pip-run -q pyopenssl -- -m easy_install requests
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/jaraco/code/main/setuptools/easy_install.py", line 5, in <module>
    main()
  File "/Users/jaraco/code/main/setuptools/setuptools/command/easy_install.py", line 2316, in main
    **kw
  File "/Users/jaraco/code/main/setuptools/setuptools/__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/jaraco/code/main/setuptools/setuptools/command/easy_install.py", line 418, in run
    self.easy_install(spec, not self.no_deps)
  File "/Users/jaraco/code/main/setuptools/setuptools/command/easy_install.py", line 667, in easy_install
    not self.always_copy, self.local_index
  File "/Users/jaraco/code/main/setuptools/setuptools/package_index.py", line 654, in fetch_distribution
    self.find_packages(requirement)
  File "/Users/jaraco/code/main/setuptools/setuptools/package_index.py", line 487, in find_packages
    self.scan_url(self.index_url + requirement.unsafe_name + '/')
  File "/Users/jaraco/code/main/setuptools/setuptools/package_index.py", line 827, in scan_url
    self.process_url(url, True)
  File "/Users/jaraco/code/main/setuptools/setuptools/package_index.py", line 348, in process_url
    f = self.open_url(url, tmpl % url)
  File "/Users/jaraco/code/main/setuptools/setuptools/package_index.py", line 766, in open_url
    return open_with_auth(url, self.opener)
  File "/Users/jaraco/code/main/setuptools/setuptools/package_index.py", line 961, in _socket_timeout
    return func(*args, **kwargs)
  File "/Users/jaraco/code/main/setuptools/setuptools/package_index.py", line 1079, in open_with_auth
    fp = opener(request)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/Users/jaraco/code/main/setuptools/setuptools/ssl_support.py", line 168, in https_open
    lambda host, **kw: pick_ssl()(host, self.ca_bundle, **kw), req
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-5_6omuyu/OpenSSL/SSL.py", line 1765, in sendall
    self._raise_ssl_error(self._ssl, result)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-5_6omuyu/OpenSSL/SSL.py", line 1614, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

[jaraco]

I'm removing the 'critical' label, as I've made substantial progress to dropping reliance on setup_requires in pypa/packaging-problems#219.

Still, it would be nice if easy_install didn't just fail on these older SSL builds.

Another option to supplementing the SSL support would be to defer to pip for installing packages (#917).

[ssbarnea]

I am wondering if the problem I am describing at tox-dev/tox#809 (comment) is not the same, because the package that fails to install is pbr, which is the only package mentioned in setup_requires, for good reasons.

This happens when I have my custom SSL proxy and while having correct cert store environment variables defined:

CURL_CA_BUNDLE=/Users/ssbarnea/cacert.pem
REQUESTS_CA_BUNDLE=/Users/ssbarnea/cacert.pem
SSL_CERT_FILE=/Users/ssbarnea/cacert.pem

Yep, all 3 needed as different tools/libraries expect different variables.

[jaraco]

setup_requires is deprecated (#2823). This issue is thus obsolete.