Google account challenge authentication fails with 'Unable to log in: HTTP Error 400: Bad Request'

[Ruin0x11]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.01.24
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

 »  youtube-dl "https://www.youtube.com/playlist?list=WL" -u [email protected] -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/playlist?list=WL', '-u', 'PRIVATE', '-v']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.01.24
[debug] Python version 3.8.1 (CPython) - Linux-5.4.14-arch1-1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[youtube:watchlater] Downloading login page
[youtube:watchlater] Looking up account info
[youtube:watchlater] Logging in
WARNING: Unable to log in: HTTP Error 400: Bad Request
[youtube:watchlater] WL: Downloading webpage
ERROR: This playlist does not exist.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 3238, in _real_extract
    _, playlist = self._extract_playlist('WL')
  File "/usr/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 2738, in _extract_playlist
    raise ExtractorError(message, expected=True)
youtube_dl.utils.ExtractorError: This playlist does not exist.

Description

When I was looking into an issue with Google account login with invidious, I tried to see if youtube-dl also had the same login issue, and it turns out it does.

When trying to log in with a Google email and password, regardless of 2FA being enabled or not, the login will always fail with the message Unable to log in: HTTP Error 400: Bad Request.

I looked into the specific requests that are being made by the auth handler. The authentication is successfully passing the lookup stage where it makes a request to https://accounts.google.com/_/signin/sl/lookup, but gets back a 400 response when making a challenge request to /_/signin/sl/challenge. What seems to have happened is Google changed something internally on this endpoint that breaks the existing authentication.

If this is a duplicate issue, the issue it is a duplicate of is not clear to me at all. In #23832 there are four separate issues that are linked to as justification that this specific issue is a duplicate of one of the others. However, the error output of each issue differs subtly from this one.

#15184: It reaches the 2FA prompt (Type 2-step verification code and press [Return]:), so the challenge request must have succeeded at that point.
#15423: The author mentions supplying an authentication code from Google Authenticator after giving a password, so this is after the challenge step.
#15926: The specific error is different (WARNING: Unable to login: Invalid password/Unable to download webpage: HTTP Error 404: Not Found)
#17216: The specific error is different (WARNING: Unable to login: Invalid password)

None of these issues contain a Unable to log in: HTTP Error 400: Bad Request error. The newest one is from August 2018 and the oldest is January 2018.

On the other hand there are numerous issues all created after April 2019 which were marked as "duplicate" with similar circumstances as this issue with little or no followup:

#23856
#23813
#23769
#23449
#22925
#22725
#21691
#21677
#21313
#21285
#21065
#21054
#20814
#20755

I do not believe this issue is a duplicate of #11270 as stated in this issue. That issue is from November 2016 and appears to be separate: the original author appeared to write about an issue related to a DNS configuration at first, but several people followed up on the issue years later around May 2019 on this seemingly unrelated issue that just happened to share the same error message. I would find it suspicious if the number of independent reports of this issue suddenly increased after April 2019 (the earliest mention of this bug) after no other similar issues since the one in 2016.

Even so the commit which closed issue #11270 (baf67a6) has not fixed this bug. I tested the same code change independently by making the requests to Google's challenge endpoint in a different language and still got a 400 response. See here for the code.

Someone mentioned that it could be solved by adding cookies to the request but I haven't verified this.

And after looking through the entire list of issues I cannot find one which is currently open that has the same reproduction of the log output. It might just be that I can't locate it myself so I would appreciate being able to know which one it is the duplicate of if there actually is one.

In reproducing this bug my account had 2FA enabled, but even after disabling it the same 400 response was returned. Thus, I do not believe this issue has anything to do with 2FA, so the PR at #18521 would probably not fix it (the PR adds code that runs after the challenge request succeeds, but in this case the challenge request is not succeeding so it will never reach the 2FA stage).

I'm not sure what's going on if so many other people are running across this issue but the issues they open are being closed with no explanation or even a cursory mention of what issue this is a duplicate of. Would it be possible to receive some kind of an explanation of what is happening with this bug? It seems that some people have been unable to download private playlists like Watch Later for almost a year due to this issue. And if you choose to close this issue as another duplicate, would it be possible to at least indicate which issue it is a duplicate of?

[flyscan]

Side note - searching for a solution to unrelated problems and the solution for adding cookies to the request worked around the HTTP Error 400 for me. Thankyou for finding that.

[qome]

And if you choose to close this issue as another duplicate, would it be possible to at least indicate which issue it is a duplicate of?

Nope.

[jsmith-github]

And if you choose to close this issue as another duplicate, would it be possible to at least indicate which issue it is a duplicate of?

Dear @dstftw ,
Do you even read what the reporter wrote?

[broberson]

Copying the cookies from both google.com and youtube.com, merging the files, and feeding that to youtube-dl works for me. Still getting the Error 400: Bad Request message, but youtube-dl does successfully log into YouTube and downloads purchased content only available after logging in with my credentials.

Perhaps the Bad Request error message is causing people to assume it's not working.

Edit: Nevermind. The resulting video plays for a few seconds and then craps out. Corrupted somehow.

[qome]

Do you even read what the reporter wrote?

Guarantee nope. Wish someone else would take over responding to bug reports or he'd remove the massive wall of text the program outputs saying to report a bug.

[Tzahi12345]

Just want to drop in and say I'm experiencing the same issue, logging in with a google account.

[saintmedusa]

Also having this issue.

[greyman56]

I am getting this issue too:
"[youtube] Logging in
WARNING: Unable to log in: HTTP Error 400: Bad Request"

[Ventriduct]

I am not able to download private playlists:

$ youtube-dl -u [email protected] -j --flat-playlist "https://www.youtube.com/playlist?list=<PLAYLIST ID>" | jq -r '.id' | sed 's_^_https://youtu.be/_' > result.log
Type account password and press [Return]:
WARNING: Unable to log in: HTTP Error 400: Bad Request
ERROR: This playlist does not exist.

[asoretmadolell]

I can confirm that this worked for me #21313 (comment)

[daregod]

Also having this issue.

same

[dirkf]

Reopening as a flag for YT login overhaul.

See also PR #18521.