Repository: encode/httpx

Issue #1434: HTTPS proxies support
Created: 2020-12-21T15:54:28Z
Refs #1424, #1428
Terminology

HTTP proxy: a proxy server which supports connecting to it via HTTP. HTTP requests are forwarded, HTTPS requests are tunneled (via HTTP CONNECT). — HTTPX has good support for those, no questions asked. ✅
HTTPS proxy: a proxy server that supports connecting to it via HTTPS. Only HTTPS requests are supported, and must (?) be tunneled (TLS-in-TLS). — This is what seems to be missing still. ❌

Problem statement
We've seen reports of issues recently such as #1424 and #1428 that reveal that our proxies implementation does not properly support HTTPS proxies yet.
My understanding right now is that supporting this requires implementing a technique known as "TLS-in-TLS" (or perhaps "nested TLS"). Here's how that works:

HTTPX issues a CONNECT request to the proxy, at https://proxy.org. This may use a dedicated proxy_ssl_context with proxy-specific certs, that I'll mark as TLS(p) ("p" as "proxy").
The proxy establishes a TCP tunnel to the target. The HTTPX-proxy "half" of the TCP connection is over TLS(p), and the other one is not TLS-enabled yet.
HTTPX must perform a TLS handshake with the target server, so that proxy-server "half" of the TCP connection becomes encrypted over TLS(t) ("t" as "target"). I'm not 100% certain I understand how that works. Right now I assume TLS handshake packets would be sent TLS(p)-encrypted to the proxy, which """decrypts""" them and sends them to the server for us. The server responds with its SERVER HELLO. The proxy """encrypts"" them back over TLS(p) and HTTPX sees them. The second pass of the handshake follows the same pattern. (I've put quotation marks because this is actually done without the proxy being able to actually intercept those packets — anyone could confirm?)

✅ Right now we can do steps 1/ and 2/, with the nuance that we have a single ssl_context option that's used for both proxy CONNECT and the handshake (we'd want to have proxy_ssl_context and ssl_context).
❌ What is definitely missing is step 3/.
Right now we attempt to do start_tls(), as if we were tunneling over a standard HTTP connection with the proxy — and generally that fais in a variety of ways depending on sync / async, async library, custom certs, HTTP/1.1 vs HTTP/2, proxy server implementation, etc.
To reproduce
Right now the following would fail:
proxies = {"https": "https://proxy.org:443"}
with httpx.Client(proxies=proxies) as client:
    response = client.get("https://example.org")
TODO: full pproxy setup (or perhaps proxy.py, which seems to support HTTPS proxying fully), full sample tracebacks.
Additional context
Marked this as "requests-compat" because this is now supported in urllib3 as of 1.26. It landed via this pull request: urllib3/urllib3#1923. AFAICT they had to implement TLS-in-TLS themselves, overriding the standard http.client connection implementation because that one doesn't support TLS-in-TLS.
Marked this as "httpcore" because our proxy implementation lives there: https://github.com/encode/httpcore

Duplicate issue #1373: httpx.ConnectError: [SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:1108)
Created: 2020-10-28T10:05:39Z
import httpx
url = 'https://www.xhssy.com.cn/forum.php?mod=forumdisplay&fid=81&filter=author&orderby=dateline'
resp = httpx.get(url)
print(resp.status_code)

import requests
url = 'https://www.xhssy.com.cn/forum.php?mod=forumdisplay&fid=81&filter=author&orderby=dateline'
resp = requests.get(url)
print(resp.status_code)

When i use  httpx.get  return exception：httpx.ConnectError: [SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:1108)，but  requests.get is ok.
`Traceback (most recent call last):
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_exceptions.py", line 326, in map_exceptions
yield
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_client.py", line 861, in _send_single_request
(status_code, headers, stream, ext) = transport.request(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpcore_sync\connection_pool.py", line 200, in request
response = connection.request(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpcore_sync\connection.py", line 87, in request
self.socket = self._open_socket(timeout)
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpcore_sync\connection.py", line 108, in _open_socket
return self.backend.open_tcp_stream(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpcore_backends\sync.py", line 144, in open_tcp_stream
return SyncSocketStream(sock=sock)
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\contextlib.py", line 131, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpcore_exceptions.py", line 12, in map_exceptions
raise to_exc(exc) from None
httpcore.ConnectError: [SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:1108)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:/python/project/yuqing_tool/httpx_requests.py", line 3, in 
resp = httpx.get(url)
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_api.py", line 181, in get
return request(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_api.py", line 93, in request
return client.request(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_client.py", line 733, in request
return self.send(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_client.py", line 767, in send
response = self._send_handling_auth(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_client.py", line 805, in _send_handling_auth
response = self._send_handling_redirects(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_client.py", line 837, in _send_handling_redirects
response = self._send_single_request(request, timeout)
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_client.py", line 861, in _send_single_request
(status_code, headers, stream, ext) = transport.request(
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\contextlib.py", line 131, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\zhs\AppData\Local\Programs\Python\Python38-32\lib\site-packages\httpx_exceptions.py", line 343, in map_exceptions
raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.ConnectError: [SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:1108)`

Closing comment (2022-04-16T14:26:12Z): EDIT: Never mind, I see this is a different issue tracked in #1434.

I'm seeing this error on httpx 0.22.0 even with a HTTPS proxy that supports TLSv1.3. The proposed workaround does not fix it.
❯ openssl s_client -connect hu-001.whiskergalaxy.com:443 </dev/null 2>&1 | grep TLS
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384

❯ python
Python 3.10.4 (main, Apr  4 2022, 14:54:08) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import httpx
>>> httpx.__version__
'0.22.0'
>>> client = httpx.Client(proxies={"all://": "https://REDACTED:REDACTED@hu-001.whiskergalaxy.com"})
>>> client.get("https://example.com")
Traceback (most recent call last):
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/backends/sync.py", line 57, in start_tls
    raise exc
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/backends/sync.py", line 52, in start_tls
    sock = ssl_context.wrap_socket(
  File "/home/nyuszika7h/.pyenv/versions/3.10.4/lib/python3.10/ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/nyuszika7h/.pyenv/versions/3.10.4/lib/python3.10/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/home/nyuszika7h/.pyenv/versions/3.10.4/lib/python3.10/ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNEXPECTED_MESSAGE] unexpected message (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
    raise exc
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
    response = connection.handle_request(request)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/_sync/http_proxy.py", line 283, in handle_request
    stream = stream.start_tls(**kwargs)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/backends/sync.py", line 49, in start_tls
    with map_exceptions(exc_map):
  File "/home/nyuszika7h/.pyenv/versions/3.10.4/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: [SSL: UNEXPECTED_MESSAGE] unexpected message (_ssl.c:997)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1025, in get
    return self.request(
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_client.py", line 802, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_client.py", line 889, in send
    response = self._send_handling_auth(
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_client.py", line 917, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_client.py", line 990, in _send_single_request
    response = transport.handle_request(request)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/home/nyuszika7h/.pyenv/versions/3.10.4/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/nyuszika7h/test/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [SSL: UNEXPECTED_MESSAGE] unexpected message (_ssl.c:997)
>>>
```