Repository: yt-dlp
Issue: Fails to utilize proxy
Labels: bug,triage,core:networking

### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

Development release of yt-dlp 2024.9.2.232855.dev0 works locally, but when deployed to cloud it constantly faces IP bans (requires to sign in to verify that it is not a bot). I tried to use Zyte proxies with yt-dlp, but it seems that it fails to utilize them.
I guess it is problem that it lacks functionality of verification of CA certificates. 

Even though it has option "--client-certificate", it did not help. Might be this option is not developed to work with CA certificates?

youtube-transcript-api had the similar issue that it failed to utilize Zyte proxies and it was resolved by this fork where verification of CA certificate was added:
https://github.com/jdepoix/youtube-transcript-api/issues/303#issuecomment-2274163459
https://github.com/danielsanmartin/youtube-transcript-api/commit/f6f51d007f14fd588a3c95c4c7abb6d36f536e4a

Could you add option to verify the CA certificates, so that it would be possible to use Zyte proxies for ban handling with the library? I tried a number of ways to use Zyte proxies without verifying zyte-ca.crt, but without the verification of zyte-ca.crt, it does not work and gives an error.

Here is Zyte’s information on their zyte-ca.crt certificate: https://docs.zyte.com/misc/ca.html


I am unsure whether this issue is a bug or a feature request, as I do not fully understand what causes it. I used the same Zyte proxy and zyte-ca.crt certificate with the 'requests' library and the mentioned youtube-transcript-api branch, where it worked. However, it fails here for some reason.


How I was trying to utilize Zyte proxy with yt-dlp:

First, I ran the Python code that contained this:
```
zyte_proxy = f"https://{zyte_api_key}:@proxy.zyte.com:8011"
zyte_certificate_path = "/path/to/zyte-ca.crt"

# yt-dlp options
ydl_opts = {
    'format': 'm4a/bestaudio/best',  
    'proxy': zyte_proxy,  # Use Zyte proxy
    '--client-certificate': zyte_certificate_path,  # Path to the Zyte certificate
    'nocheckcertificate': False,  # Ensure certificates are verified
    'postprocessors': [{  # Extract audio using ffmpeg
        'key': 'FFmpegExtractAudio',
        'preferredcodec': 'm4a',
    }],
}
```

I receive in terminal this:
"

> Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP.

"

Then I update zyte_proxy URL to be HTTP, and run again, and receive in terminal this:
"

> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

"

Then I remove options "nocheckcertificate" and "--client-certificate" and run again, and receive this in terminal: "

> Unable to download webpage: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

"

Then I update zyte_proxy URL to HTTPS and run again, and receive this:
"

> Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP.

"

Also, when I ran command "yt-dlp -U" in terminal it prints:
"

> Latest version: nightly@2024.09.02.232855 from yt-dlp/yt-dlp-nightly-builds
> yt-dlp is up to date (nightly@2024.09.02.232855 from yt-dlp/yt-dlp-nightly-builds)

"


I also added verbose output of this code:
```
zyte_proxy = f"https://{zyte_api_key}:@proxy.zyte.com:8011"
zyte_certificate_path = "/path/to/zyte-ca.crt"

# yt-dlp options
ydl_opts = {
    'format': 'm4a/bestaudio/best',  
    'verbose': True, 
    'proxy': zyte_proxy,  # Use Zyte proxy
    '--client-certificate': zyte_certificate_path,  # Path to the Zyte certificate
    'nocheckcertificate': False,  # Ensure certificates are verified
    'postprocessors': [{  # Extract audio using ffmpeg
        'key': 'FFmpegExtractAudio',
        'preferredcodec': 'm4a',
    }],
}
```

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version nightly@2024.09.02.232855 from yt-dlp/yt-dlp-nightly-builds [e8e6a982a] (pip) API
[debug] params: {'format': 'm4a/bestaudio/best', 'verbose': True, 'proxy': 'https://I_hided_zyte_api_key_here:@proxy.zyte.com:8011', '--client-certificate': '/path/to/zyte-ca.crt', 'nocheckcertificate': False, 'postprocessors': [{'key': 'FFmpegExtractAudio', 'preferredcodec': 'm4a'}], 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.10.14 (CPython arm64 64bit) - macOS-14.5-arm64-arm-64bit (OpenSSL 3.3.1 4 Jun 2024)
[debug] exe versions: ffmpeg 7.0.2 (setts), ffprobe 7.0.2
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.08.30, mutagen-1.47.0, requests-2.32.3, sqlite3-3.46.0, urllib3-2.2.2, websockets-13.0.1
[debug] Proxy map: {'all': 'https://I_hided_zyte_api_key_here:@proxy.zyte.com:8011'}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1832 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=2WcbPcGrQZU
[youtube] 2WcbPcGrQZU: Downloading webpage
WARNING: [youtube] Unable to download webpage: ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)')))
[youtube] 2WcbPcGrQZU: Downloading ios player API JSON
[youtube] 2WcbPcGrQZU: Downloading ios player API JSON
WARNING: [youtube] ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))). Retrying (1/3)...
WARNING: [youtube] ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))). Retrying (2/3)...
[youtube] 2WcbPcGrQZU: Downloading ios player API JSON
WARNING: [youtube] ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))). Retrying (3/3)...
[youtube] 2WcbPcGrQZU: Downloading ios player API JSON
WARNING: [youtube] Unable to download API page: ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))) (caused by ProxyError("('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)')))")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[youtube] 2WcbPcGrQZU: Downloading iframe API JS
WARNING: [youtube] Unable to download webpage: ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)')))
[youtube] 2WcbPcGrQZU: Downloading web creator player API JSON
WARNING: [youtube] ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))). Retrying (1/3)...
[youtube] 2WcbPcGrQZU: Downloading web creator player API JSON
[youtube] 2WcbPcGrQZU: Downloading web creator player API JSON
WARNING: [youtube] ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))). Retrying (2/3)...
WARNING: [youtube] ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))). Retrying (3/3)...
[youtube] 2WcbPcGrQZU: Downloading web creator player API JSON
WARNING: [youtube] Unable to download API page: ('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)'))) (caused by ProxyError("('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1007)')))")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR: [youtube] 2WcbPcGrQZU: Failed to extract any player response; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 740, in extract
    ie_result = self._real_extract(url)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/youtube.py", line 4263, in _real_extract
    webpage, master_ytcfg, player_responses, player_url = self._download_player_responses(url, smuggled_data, video_id, webpage_url)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/youtube.py", line 4227, in _download_player_responses
    player_responses, player_url = self._extract_player_responses(
  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/youtube.py", line 3904, in _extract_player_responses
    raise ExtractorError('Failed to extract any player response')

Traceback (most recent call last):
  File "/path/to/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1626, in wrapper
    return func(self, *args, **kwargs)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1761, in __extract_info
    ie_result = ie.extract(url)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 740, in extract
    ie_result = self._real_extract(url)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/youtube.py", line 4263, in _real_extract
    webpage, master_ytcfg, player_responses, player_url = self._download_player_responses(url, smuggled_data, video_id, webpage_url)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/youtube.py", line 4227, in _download_player_responses
    player_responses, player_url = self._extract_player_responses(
  File "/path/to/lib/python3.10/site-packages/yt_dlp/extractor/youtube.py", line 3904, in _extract_player_responses
    raise ExtractorError('Failed to extract any player response')
yt_dlp.utils.ExtractorError: [youtube] 2WcbPcGrQZU: Failed to extract any player response; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/te_yt_dlp_with_proxy_and_certificate_2.py", line 29, in <module>
    error_code = ydl.download(URLS)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3607, in download
    self.__download_wrapper(self.extract_info)(
  File "/path/to/Documents/miniconda3/envs/dev_conda_based_no_captions_files_video_summary_1/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3582, in wrapper
    res = func(*args, **kwargs)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1615, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1644, in wrapper
    self.report_error(str(e), e.format_traceback())
  File "/path/to/Documents/miniconda3/envs/dev_conda_based_no_captions_files_video_summary_1/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1092, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "/path/to/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1031, in trouble
    raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: [youtube] 2WcbPcGrQZU: Failed to extract any player response; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```