TypeError: __init__() got an unexpected keyword argument 'check_hostname'

[ReenigneArcher]

Checklist

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2021.12.17
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

tests\unit\test_youtube_dl_helper.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Contents\Code\youtube_dl_helper.py:49: in process_youtube
    download=False  # We just want to extract the info
venv\lib\site-packages\youtube_dl\YoutubeDL.py:852: in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
venv\lib\site-packages\youtube_dl\YoutubeDL.py:859: in wrapper
    return func(self, *args, **kwargs)
venv\lib\site-packages\youtube_dl\YoutubeDL.py:955: in __extract_info
    ie_result = ie.extract(url)
venv\lib\site-packages\youtube_dl\extractor\common.py:565: in extract
    ie_result = self._real_extract(url)
venv\lib\site-packages\youtube_dl\extractor\youtube.py:1850: in _real_extract
    webpage_url + '&bpctr=9999999999&has_verified=1', video_id, fatal=False)
venv\lib\site-packages\youtube_dl\extractor\common.py:832: in _download_webpage
    expected_status=expected_status)
venv\lib\site-packages\youtube_dl\extractor\common.py:698: in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
venv\lib\site-packages\youtube_dl\extractor\common.py:665: in _request_webpage
    return self._downloader.urlopen(url_or_request)
venv\lib\site-packages\youtube_dl\YoutubeDL.py:2461: in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
venv\lib\site-packages\future\backports\urllib\request.py:500: in open
    response = self._open(req, data)
venv\lib\site-packages\future\backports\urllib\request.py:518: in _open
    '_open', req)
venv\lib\site-packages\future\backports\urllib\request.py:472: in _call_chain
    result = func(*args)
venv\lib\site-packages\youtube_dl\utils.py:2889: in https_open
    req, **kwargs)
venv\lib\site-packages\future\backports\urllib\request.py:1261: in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ydl_handler = <youtube_dl.utils.YoutubeDLHTTPSHandler object at 0x0000000007C37C88>
http_class = <class httplib.HTTPSConnection at 0x00000000042D62E8>
is_https = True, args = ('www.youtube.com',)
kwargs = {'check_hostname': None, 'context': <ssl.SSLContext object at 0x0000000007BBC7B8>, 'strict': True, 'timeout': 600}

    def _create_http_connection(ydl_handler, http_class, is_https, *args, **kwargs):
        # Working around python 2 bug (see http://bugs.python.org/issue17849) by limiting
        # expected HTTP responses to meet HTTP/1.0 or later (see also
        # https://github.com/ytdl-org/youtube-dl/issues/6727)
        if sys.version_info < (3, 0):
            kwargs['strict'] = True
>       hc = http_class(*args, **compat_kwargs(kwargs))
E       TypeError: __init__() got an unexpected keyword argument 'check_hostname'

venv\lib\site-packages\youtube_dl\utils.py:2524: TypeError
---------------------------- Captured stdout call -----------------------------
[youtube] dQw4w9WgXcQ: Downloading webpage
============================== warnings summary ===============================
tests/unit/test_youtube_dl_helper.py::test_process_youtube
  C:\Users\ReenigneArcher\Documents\GitHub\LizardByte\Themerr-plex\venv\lib\site-packages\future\backports\urllib\request.py:359: DeprecationWarning: Request.is_unverifiable method is deprecated.
    warnings.warn(msg, DeprecationWarning, stacklevel=1)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========== 1 failed, 7 passed, 1 skipped, 1 warnings in 4.80 seconds ==========

Description

I am using youtube-dl as a python library. In this case, I am only attempting to download the data for a video.

    youtube_dl_params = dict(
        outmpl='%(id)s.%(ext)s',
        youtube_include_dash_manifest=False,
    )

    ydl = youtube_dl.YoutubeDL(params=youtube_dl_params)

    with ydl:
        result = ydl.extract_info(
            url=url,
            download=False  # We just want to extract the info
        )
        if 'entries' in result:
            # Can be a playlist or a list of videos
            video_data = result['entries'][0]
        else:
            # Just a video
            video_data = result

This is the main error I am receiving when running my tests in pytest.

E       TypeError: __init__() got an unexpected keyword argument 'check_hostname'

Which corresponds to the def _create_http_connection(ydl_handler, http_class, is_https, *args, **kwargs): function in utils.py.

The same issue happens on commit 26035bd and commit 66ab081.

This is my test video. https://www.youtube.com/watch?v=dQw4w9WgXcQ

I am on Python 2.7, and from what I can tell the _check_hostname is not supposed to be used unless it is Python 3.x.

I have determined this is occurring after a new dependency was added which has the following line (for Python 2.7 compatibility). What is interesting is the portion of the code failing the test is not even related to this new library, but simply importing the library anywhere before the test run causes this error.

from future import standard_library
standard_library.install_aliases()

Given that this library is supposed to support Python 2.7 (and 2.6?) would it be possible to support this install_aliases option? In general this makes it a lot simpler to support Python 2.7 and 3.x.

https://python-future.org/_modules/future/standard_library.html#install_aliases

[dirkf]

Seems like this library sets up imports that aren't expected in Py2, so the compat code thinks it's in Py3.

As the linked page says:

This is a cleaner alternative to this idiom (see
http://docs.pythonsprints.com/python3_porting/py-porting.html):

   try:
       import queue
   except ImportError:
       import Queue as queue

So an answer could be to import the compat module before standard_library.install_aliases().

If the __future__.standard_library module had existed way back when, yt-dl would perhaps have used it instead of the home-grown compat.

[ReenigneArcher]

This worked for now. Thank you!

from youtube_dl import compat  # noqa: F401

I will have to spend some time re-writing the backport to not use the install_aliases() method.

[dirkf]

Hmmm. I had to back to the docs to confirm how imports are supposed to work.

It seems that when a module is imported, it's loaded and cached in sys.modules. Subsequent references to, and imports of, names in the module don't cause their definition to be evaluated again.

So importing the compat module before standard_library.install_aliases() means that the names exported from it are generally correctly defined for the Python version.

However, some code in yt-dl may use the idiom shown above that are not evaluated when importing compat:

• code in function definitions in the compat module
• code in other modules.

Such code may still be confused by the installed aliases.

... yt-dl would perhaps have used [__future__.standard_library] instead of the home-grown compat.

But only if the project licences could have been made compatible.

[ReenigneArcher]

It seems that when a module is imported, it's loaded and cached in sys.modules. Subsequent references to, and imports of, names in the module don't cause their definition to be evaluated again.

Yes, I believe that is correct.

I think the safest approach is that I need to re-write the imports for my backport of python-plexapi. I did not expect that the install_aliases() would globally change Python 2's interpretation... I assumed it was on a per file basis. But yea, this is definitely isn't good as I can't guarantee that everything I import will be compatible.