Repository: youtube-dl
Issue: Google account challenge authentication fails with 'Unable to log in: HTTP Error 400: Bad Request'
Labels: duplicate

<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.01.24. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
- Finally, put x into all relevant boxes (like this [x])
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running youtube-dl version **2020.01.24**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] youtube-dl version 2020.01.24
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
 »  youtube-dl "https://www.youtube.com/playlist?list=WL" -u ipickering2@gmail.com -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/playlist?list=WL', '-u', 'PRIVATE', '-v']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.01.24
[debug] Python version 3.8.1 (CPython) - Linux-5.4.14-arch1-1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[youtube:watchlater] Downloading login page
[youtube:watchlater] Looking up account info
[youtube:watchlater] Logging in
WARNING: Unable to log in: HTTP Error 400: Bad Request
[youtube:watchlater] WL: Downloading webpage
ERROR: This playlist does not exist.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 3238, in _real_extract
    _, playlist = self._extract_playlist('WL')
  File "/usr/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 2738, in _extract_playlist
    raise ExtractorError(message, expected=True)
youtube_dl.utils.ExtractorError: This playlist does not exist.
```


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

When I was looking into an issue with Google account login with [invidious](https://github.com/omarroth/invidious/issues/754), I tried to see if `youtube-dl` also had the same login issue, and it turns out it does.

When trying to log in with a Google email and password, regardless of 2FA being enabled or not, the login will always fail with the message `Unable to log in: HTTP Error 400: Bad Request`.

I looked into the specific requests that are being made by the auth handler. The authentication is successfully passing the lookup stage where it makes a request to `https://accounts.google.com/_/signin/sl/lookup`, but gets back a 400 response when making a challenge request to `/_/signin/sl/challenge`. What seems to have happened is Google changed something internally on this endpoint that breaks the existing authentication.

If this is a duplicate issue, the issue it is a duplicate of is not clear to me at all. In #23832 there are four separate issues that are linked to as justification that this specific issue is a duplicate of one of the others. However, the error output of each issue differs subtly from this one.

#15184: It reaches the 2FA prompt (`Type 2-step verification code and press [Return]:`), so the `challenge` request must have succeeded at that point.
#15423: The author mentions supplying an authentication code from Google Authenticator after giving a password, so this is after the `challenge` step.
#15926: The specific error is different (`WARNING: Unable to login: Invalid password`/`Unable to download webpage: HTTP Error 404: Not Found`)
#17216: The specific error is different (`WARNING: Unable to login: Invalid password`)

None of these issues contain a `Unable to log in: HTTP Error 400: Bad Request` error. The newest one is from August 2018 and the oldest is January 2018.

On the other hand there are numerous issues all created after April 2019 which were marked as "duplicate" with similar circumstances as this issue with little or no followup:

#23856
#23813
#23769
#23449
#22925
#22725
#21691
#21677
#21313
#21285
#21065
#21054
#20814
#20755

I do not believe this issue is a duplicate of #11270 as stated in [this issue](https://github.com/ytdl-org/youtube-dl/issues/20755#issuecomment-489716927). That issue is from November 2016 and appears to be separate: the original author appeared to write about an issue related to a DNS configuration at first, but several people followed up on the issue years later around May 2019 on this seemingly unrelated issue that just happened to share the same error message. I would find it suspicious if the number of independent reports of this issue suddenly increased after April 2019 (the earliest mention of this bug) after no other similar issues since the one in 2016.

Even so the commit which closed issue #11270 (baf67a6) has not fixed this bug. I tested the same code change independently by making the requests to Google's challenge endpoint in a different language and still got a 400 response. See [here](https://gist.github.com/Ruin0x11/a32c4ba33d386b51797a3719ad2c14e4) for the code.

Someone mentioned that it could be solved by [adding cookies to the request](https://github.com/ytdl-org/youtube-dl/issues/21313#issuecomment-499496235) but I haven't verified this.

And after looking through the entire list of issues I cannot find one which is currently open that has the same reproduction of the log output. It might just be that I can't locate it myself so I would appreciate being able to know which one it is the duplicate of if there actually is one.

In reproducing this bug my account had 2FA enabled, but even after disabling it the same 400 response was returned. Thus, I do not believe this issue has anything to do with 2FA, so the PR at #18521 would probably not fix it (the PR adds code that runs after the `challenge` request succeeds, but in this case the `challenge` request is not succeeding so it will never reach the 2FA stage).

I'm not sure what's going on if so many other people are running across this issue but the issues they open are being closed with no explanation or even a cursory mention of what issue this is a duplicate of. *Would it be possible to receive some kind of an explanation of what is happening with this bug?* It seems that some people have been unable to download private playlists like Watch Later for almost a year due to this issue. And if you choose to close this issue as another duplicate, would it be possible to at least indicate which issue it is a duplicate of?