Chromium Spelunking: Connecting to Proxies
Tags: cpp, chromium
Published: 2023-09-28 20:39:57

Having [successfully fetched a URL](https://dev.to/djmitche/chromium-spelunking-the-io-thread-464n) with the Chromium network stack, it's time to return to the reason I began this journey: how does Chromium connect to proxies?

## Proxy Review

First, a quick review of HTTP proxying.

In the beginning, there was a simple protocol to proxy an HTTP transaction: connect to a proxy and, instead of sending just a path in the request line, send the entire URL. For example:
```
GET http://httbin.org/uuid HTTP/1.1
```
The proxy server then makes an outbound connection to `httpbin.org`, sends `GET /uuid HTTP/1.1` with the headers supplied by the client, and then relays the response back to the client.

This sort of proxy exposes all of the details of the transaction to the proxy (or to anyone, if not using TLS for the connection to the proxy). If the origin URL has scheme `https`, the _proxy_ will use TLS to connect to the origin, but will still handle the transaction content in cleartext. This has some obvious downsides, but an advantage is that the proxy can cache responses, saving bandwidth. This was a popular use of proxies in the aughts, but is far less common now that bandwidth is cheap and privacy is important.

An alternative method, `CONNECT`, was defined [about 25 years ago](https://datatracker.ietf.org/doc/html/rfc2817). It creates a "tunnel" through the proxy which can carry arbitrary data to and from another host. The request looks like
```
CONNECT httpbin.org:443 HTTP/1.1
```
The proxy server makes a TCP connection to the given host and port, sends back a response header, and then forwards bytes bidirectionally without further analyzing them. In fact, that protocol doesn't have to be HTTP (but it must use TCP, which will become important later).

To test proxying,I'm using [tinyproxy](https://tinyproxy.github.io/), running a very simple config on port 8080. This supports SPDY (HTTP/2), which is a complication I don't really want to consider at this point, but the analysis ends up quite similar to HTTP/1.

## Configuring a Proxy

So, how does the network service decide to use a proxy for a request? It's complicated.

Let's start at the bottom. [`net::ProxyServer`](https://source.chromium.org/chromium/chromium/src/+/main:net/base/proxy_server.h) defines an actual server to connect to. It has a scheme, host, and port. One of those schemes is "DIRECT" which means do not use a proxy. The others include HTTP, HTTPS, SOCKS, and QUIC, and just describe how to connect to the proxy server.

[`net::ProxyList`](https://source.chromium.org/chromium/chromium/src/+/main:net/proxy_resolution/proxy_list.h) represents a list of `ProxyServer` instances and handles fallback from one to the next. This allows, for example, an enterprise to configure traffic to go via a local proxy but fall back to DIRECT when that proxy is down, such as when using a caching proxy.

Scoping out another level, [`net::ProxyConfig`](https://source.chromium.org/chromium/chromium/src/+/main:net/proxy_resolution/proxy_config.h) represents a configuration for when to use which proxies. This can be manually configured or can refer to a [PAC script](https://developer.mozilla.org/en-US/docs/Web/HTTP/Proxy_servers_and_tunneling/Proxy_Auto-Configuration_PAC_file) that defines the configuration.

The `ProxyConfig` comes from a [`net::ProxyConfigService`](https://source.chromium.org/chromium/chromium/src/+/main:net/proxy_resolution/proxy_config_service.h?q=net::ProxyConfigService), which is used by a [`net::ProxyResolutionService`](https://source.chromium.org/chromium/chromium/src/+/main:net/proxy_resolution/configured_proxy_resolution_service.h) to determine the `ProxyList` to use for a given URL. There's a lot of complexity here that we won't get into, including auto-configuration of proxies and downloading and executing PAC scripts, but the end result is a `ProxyList`.

For `churl`, we want to hard-code a proxy, so I updated the `ProxyConfigService` I defined [earlier in this series](https://dev.to/djmitche/chromium-spelunking-churl-5dof) to return a config pointing to a proxy server:

```c++
class ProxyConfigServiceHardCoded : public net::ProxyConfigService {
 public:
  // ProxyConfigService implementation:
  void AddObserver(Observer* observer) override {}
  void RemoveObserver(Observer* observer) override {}
  net::ProxyConfigService::ConfigAvailability GetLatestProxyConfig(
      net::ProxyConfigWithAnnotation* config) override {
    auto traffic_annotation = kHardCodedProxyTrafficAnnotation;
    auto proxy_config = net::ProxyConfig::CreateDirect();
    auto& proxy_rules = proxy_config.proxy_rules();
    proxy_rules.ParseFromString("localhost:8080");
    *config = net::ProxyConfigWithAnnotation(proxy_config, traffic_annotation);
    return CONFIG_VALID;
  }
};
```

## Tracing a Simple Request

OK, let's see how this works. My strategy for this sort of investigation is to add lots of debugging output -- at the beginning of each relevant function, and sometimes at key points in longer functions. Then I can follow execution within the source, comparing to the debugging output. I prefer this approach over using a debugger like GDB because I find it more efficient. Use the tools you prefer!

I'll be making a request to `https://ip.cow.org` using tinyproxy running on `http://localhost:8080`. Because the origin URL is `https`, this should use `CONNECT`.

### HTTP Cache

We saw in previous posts that the URLRequest ends up using an `HttpTransactionFactory::CreateTransaction` to create an `HttpCache::Transaction`, and starting it. That class has a rather large number of states, but adding some logging in `DoLoop` shows the sequence of states (formatted to fit your screen):

```
HttpCache::Transaction -> STATE_GET_BACKEND
HttpCache::Transaction -> STATE_GET_BACKEND_COMPLETE
HttpCache::Transaction -> STATE_INIT_ENTRY
HttpCache::Transaction -> STATE_OPEN_OR_CREATE_ENTRY
HttpCache::Transaction -> STATE_OPEN_OR_CREATE_ENTRY_COMPLETE
HttpCache::Transaction -> STATE_ADD_TO_ENTRY
HttpCache::Transaction -> STATE_ADD_TO_ENTRY_COMPLETE
HttpCache::Transaction -> STATE_SEND_REQUEST
[0925/201309.849178:ERROR:churl_bin.cc(89)] OnConnected
HttpCache::Transaction -> STATE_SEND_REQUEST_COMPLETE
HttpCache::Transaction -> STATE_FINISH_HEADERS
HttpCache::Transaction -> STATE_SUCCESSFUL_SEND_REQUEST
HttpCache::Transaction -> STATE_OVERWRITE_CACHED_RESPONSE
HttpCache::Transaction -> STATE_CACHE_WRITE_RESPONSE
HttpCache::Transaction -> STATE_CACHE_WRITE_RESPONSE_COMPLETE
HttpCache::Transaction -> STATE_TRUNCATE_CACHED_DATA
HttpCache::Transaction -> STATE_TRUNCATE_CACHED_DATA_COMPLETE
HttpCache::Transaction -> STATE_PARTIAL_HEADERS_RECEIVED
HttpCache::Transaction -> STATE_FINISH_HEADERS
HttpCache::Transaction -> STATE_FINISH_HEADERS_COMPLETE
[0925/201310.106633:ERROR:churl_bin.cc(165)] OnResponseStarted
[0925/201310.106669:ERROR:churl_bin.cc(171)] Got HTTP response code 200
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE_COMPLETE
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE_COMPLETE
```
So this provides a map to focus on what's going on in this particular request. Looking at the code, the backend- and entry-related items are just looking for values in the cache, of which there are none. The `OnConnected` callback in the URLRequest delegate occurs during the `STATE_SEND_REQUEST` segment. So let's dig in there.

`HttpCache::Transaction::DoStartRequest` calls `cache_->network_layer_->CreateTransaction(..)`. That `network_layer_` is another `HttpTransactionFactory`. The [code-search](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_transaction_factory.h;l=20;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;bpv=1;bpt=1) for that class shows a few classes that extend it, and a little debug printing reveals that this layer is an [`HttpNetworkLayer`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_network_layer.h;l=21;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;bpv=1;bpt=1), and `CreateTransaction` returns an [`HttpNetworkTransaction`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_network_transaction.h;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;bpv=1;bpt=1;l=50).

### HTTP Network Layer

This class also has a large collection of states, but the same trick helps us find the right one:
```
HttpNetworkTransaction -> STATE_NOTIFY_BEFORE_CREATE_STREAM
HttpNetworkTransaction -> STATE_CREATE_STREAM        
HttpNetworkTransaction -> STATE_CREATE_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_CONNECTED_CALLBACK     
[0925/205926.726872:ERROR:churl_bin.cc(89)] OnConnected
HttpNetworkTransaction -> STATE_CONNECTED_CALLBACK_COMPLETE
HttpNetworkTransaction -> STATE_INIT_STREAM
HttpNetworkTransaction -> STATE_INIT_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_GENERATE_PROXY_AUTH_TOKEN
HttpNetworkTransaction -> STATE_GENERATE_PROXY_AUTH_TOKEN_COMPLETE
HttpNetworkTransaction -> STATE_GENERATE_SERVER_AUTH_TOKEN
HttpNetworkTransaction -> STATE_GENERATE_SERVER_AUTH_TOKEN_COMPLETE
HttpNetworkTransaction -> STATE_INIT_REQUEST_BODY
HttpNetworkTransaction -> STATE_INIT_REQUEST_BODY_COMPLETE
HttpNetworkTransaction -> STATE_BUILD_REQUEST 
HttpNetworkTransaction -> STATE_BUILD_REQUEST_COMPLETE
HttpNetworkTransaction -> STATE_SEND_REQUEST
HttpNetworkTransaction -> STATE_SEND_REQUEST_COMPLETE
HttpNetworkTransaction -> STATE_READ_HEADERS
HttpNetworkTransaction -> STATE_READ_HEADERS_COMPLETE
[0925/225438.616076:ERROR:churl_bin.cc(165)] OnResponseStarted
[0925/225438.616113:ERROR:churl_bin.cc(171)] Got HTTP response code 200
HttpNetworkTransaction -> STATE_READ_BODY
HttpNetworkTransaction -> STATE_READ_BODY_COMPLETE
HttpNetworkTransaction -> STATE_READ_BODY
HttpNetworkTransaction -> STATE_READ_BODY_COMPLETE
```

The `STATE_CONNECTED_CALLBACK` is just calling the `OnConnected` callback. The more interesting bit is in the states before that, `STATE_CREATE_STREAM(_COMPLETE)`. The important bit of this state seems to be calling [`HttpStreamFactory::RequestStream`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_stream_factory.h;l=80). This calls back through a delegate method named `OnStreamReady`, and in this case `HttpStreamFactory` itself is the delegate. Adding a debug print in its `OnStreamReady` method shows that `used_proxy_info` includes `localhost:8080`, so that stream involves the proxy.

### HTTP Stream

The `HttpStreamFactory` class has three nested helper classes: `JobFactory`, `Job`, and `JobController`. The [`HttpStreamFactory::JobFactory`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_stream_factory_job.h;l=493?q=HttpStreamFactory::JobFactory&ss=chromium%2Fchromium%2Fsrc) class is trivial: it has a `CreateJob` method that calls the `Job` constructor. I suspect this was done as a kind of [dependency injection](https://en.wikipedia.org/wiki/Dependency_injection) to support testing the `JobController`.

[`HttpStreamFactory::JobController`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_stream_factory_job_controller.h;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;l=33) is a bit more interesting: it has a small state machine that simply resolves the proxy and then creates some jobs. The proxy resolution simply calls the `ProxyResolutionService` described above. Some debug prints confirm that this returns a `ProxyInfo` containing `localhost:8080`.

The job controller manages several jobs that run in parallel, implementing the "happy eyeballs" that I mentioned in the [Life and Times](https://dev.to/djmitche/chromium-spelunking-life-and-times-536b) post. All of these jobs are implemented with the same class. I would have expected different job subclasses per job type. Anyway, since we're not using QUIC or pre-connecting or any of that stuff, we'll just focus on the "MAIN" job.

Among many parameters, the [`HttpStreamFactory::Job`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_stream_factory_job.h;l=54?q=HttpStreamFactory::JobFactory&ss=chromium%2Fchromium%2Fsrc) constructor takes a `ProxyInfo`, so we can look for where that is used.

```
HttpStreamFactory::Job -> STATE_START                                                                                                  
HttpStreamFactory::Job -> STATE_WAIT
HttpStreamFactory::Job -> STATE_WAIT_COMPLETE
HttpStreamFactory::Job -> STATE_INIT_CONNECTION
HttpStreamFactory::Job -> STATE_INIT_CONNECTION_COMPLETE
HttpStreamFactory::Job -> STATE_CREATE_STREAM
HttpStreamFactory::Job -> STATE_CREATE_STREAM_COMPLETE
```

The `STATE_WAIT(_COMPLETE)` states are related to the job controller's coordination of multiple parallel jobs. The interesting bit is `STATE_INIT_CONNECTION`, in the [`HttpStreamFactory::Job::DoInitConnectionImpl`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_stream_factory_job.cc;l=740;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;bpv=1;bpt=1) method. This method embodies dozens of concerns -- in my opinion this a perfect example of how not to implement something like this. But, ignoring QUIC, SPDY, WebSockets, TLS, PRECONNECT, and all the rest, it comes down to a call to `InitSocketHandleForHttpRequest`, passing along the `ProxyInfo` and a plethora of additional arguments.

Let's take a moment here to notice the shift from deeply nested Java-style factories and controllers to a plain old C-style function. There's probably some interesting history here, perhaps in who wrote which bits of this code, or when they were written.

### HTTP Connection Pools

[`InitSocketHandleForHttpRequest`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/client_socket_pool_manager.cc?q=InitSocketHandleForHttpRequest), or more accurately `InitSocketPoolHelper`, gets a pool from the current `HttpNetworkSession` with `session->GetSocketPool(socket_pool_type, proxy_info.proxy_server())`. In this case the `socket_pool_type` is `NORMAL_SOCKET_POOL`, so this amounts to a call to [`ClientSocketPoolManagerImpl::GetSocketPool`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/client_socket_pool_manager_impl.cc;l=59;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;bpv=1;bpt=1) passing the proxy server through which the connection should be made (which might be `ProxyServer::Direct()` when not using a proxy). The function also creates a `ClientSocketPool::GroupId` built from the endpoint URL (`ip.cow.org` in this case) and a few partitioning parameters.

Summarizing, then, the `HttpNetworkSession` stores a connection pool for each proxy server (including direct), and within each pool indexes connections by group ID.

When a socket in a socket pool is claimed, that claim is represented by a `ClientSocketHandle`, an empty instance of which is among the parameters to `InitSocketHandleForHttpRequest`, which calls [`ClientSocketHandle::Init(..)`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/client_socket_handle.cc?q=ClientSocketHandle::Init).

This `Init` method calls the pool's `RequestSocket` method. There are two implementations of this method, one of which is for WebSockets, so in this case we're calling [`TransportClientSocketPool::RequestSocket`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/transport_client_socket_pool.cc;l=244;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;bpv=1;bpt=1) and on to `TransportClientSocketPool::RequestSocketInternal`. Assuming that there are no existing connections in the pool, and there are free slots to create new connections, this makes a new connection.

### Creating a Connection

This occurs with another set of jobs and job factories, this time with subclasses.

[`ClientSocketPool::CreateConnectJob`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/client_socket_pool.cc;l=171?q=ClientSocketPool::CreateConnectJob) uses a [`ConnectJobFactory`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/connect_job_factory.h;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;l=34) to create a [`ConnectJob`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/connect_job.h;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;l=124), passing along the origin URL (`endpoint`) and proxy server.

[`ConnectJobFactory::CreateConnectJob`](https://source.chromium.org/chromium/chromium/src/+/main:net/socket/connect_job_factory.cc;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;l=154) examines the proxy server and, in the case that it's not direct (and HTTP-like, not SOCKS) defers to an [`HttpProxyConnectJob::Factory`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_proxy_connect_job.h;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;l=92), which simply creates an [`HttpProxyConnectJob`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_proxy_connect_job.h;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;bpv=1;bpt=1;l=89), a subclass of `ConnectJob`. This, too, has a large set of states, although only a few are used in this situation:

```
HttpProxyConnectJob -> STATE_BEGIN_CONNECT
HttpProxyConnectJob -> STATE_TRANSPORT_CONNECT
HttpProxyConnectJob -> STATE_TRANSPORT_CONNECT_COMPLETE                                                                                                                                                             
HttpProxyConnectJob -> STATE_HTTP_PROXY_CONNECT
HttpProxyConnectJob -> STATE_HTTP_PROXY_CONNECT_COMPLETE
```

Checking the implementation of those states, `STATE_TRANSPORT_CONNECT` involves creating a `TransportConnectJob` (since the connection to `localhost:8080` is not using HTTPS). But at this point my head is starting to spin at the number of nested "jobs", so I'll stop here and assume that `TransportConnectJob::Connect` does what it says on the tin: connects to the host (the proxy server) specified in the [`HttpProxySocketParams`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_proxy_connect_job.h;drc=58b8aff108cb11d3755f4cc18efa10f971be9682;l=43).

### Initializing the Connection

The next state, `STATE_HTTP_PROXY_CONNECT`, wraps the socket returned from the `TransportConnectJob` in an [`HttpProxyClientSocket`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_proxy_client_socket.h;l=38?q=HttpProxyClientSocket) and calls its `Connect` method. And no surprise, there's another state machine here:

```
HttpProxyClientSocket -> STATE_GENERATE_AUTH_TOKEN
HttpProxyClientSocket -> STATE_GENERATE_AUTH_TOKEN_COMPLETE
HttpProxyClientSocket -> STATE_SEND_REQUEST
HttpProxyClientSocket -> STATE_SEND_REQUEST_COMPLETE
HttpProxyClientSocket -> STATE_READ_HEADERS
HttpProxyClientSocket -> STATE_READ_HEADERS_COMPLETE                                                                                                                                                                                                                                                                                                                                                                                                                                                     
```

We're not using proxy authentication (which is an additional complication sprinkled evenly over this entire stack!), so the interesting state here is `STATE_SEND_REQUEST`. [This](https://source.chromium.org/chromium/chromium/src/+/main:net/http/http_proxy_client_socket.cc;l=337?q=net%2Fhttp%2Fhttp_proxy_client_socket.cc) calls out to the `ProxyDelegate`, if one is configured, and then calls [`ProxyClientSocket::BuildTunnelRequest`](https://source.chromium.org/chromium/chromium/src/+/main:net/http/proxy_client_socket.cc;drc=f3ee53147a677fb8d5fac5a83e68a41041c66758;l=25) which _finally_ does something recognizable: creates a "CONNECT" request line, with the host and port for the endpoint (so, `CONNECT ip.cow.org:443 HTTP/1.1` in this example).

The next state is `STATE_READ_HEADERS`, which reads the response from the proxy. If that's a 200 OK, then the socket is connected _through_ the proxy and to the endpoint, and from here on out can be treated just like a socket connected directly to the endpoint.

### Re-Surfacing

So, let's trace the result back up through the stack. The interleaving of the logging added above helps quite a bit here:

```
HttpProxyConnectJob -> STATE_HTTP_PROXY_CONNECT_COMPLETE
HttpStreamFactory::Job -> STATE_INIT_CONNECTION_COMPLETE
HttpStreamFactory::Job -> STATE_CREATE_STREAM
HttpStreamFactory::Job -> STATE_CREATE_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_CREATE_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_CONNECTED_CALLBACK
[0926/173834.596927:ERROR:churl_bin.cc(89)] OnConnected
```

`STATE_HTTP_PROXY_CONNNECT_COMPLETE` calls the parent class's `SetSocket` to use the socket prepared earlier.

`HttpStreamFactory::Job` gets the result wrapped in a `ClientSocketHandle`. As always, it handles a half-dozen concerns in `STATE_INIT_CONNECTION_COMPLETE`, then wraps that in an `HttpBasicStream` in `STATE_CREATE_STREAM`.

The `HttpNetworkTransaction` `STATE_CREATE_STREAM_COMPLETE` then calls the `OnConnected` callback, which results in a debug log message in `churl`.

From that point, there's no further special handling of proxies -- this is a socket carrying an HTTP stream, like any other.

## What's Next

This will be the last post on this topic -- I've learned the things I wanted to learn already. However, there are certainly more things to explore:

 * What happens when making a simple proxy request, rather than tunneled?
 * What happens when a proxy tunnel fails?
 * What happens when a proxy requires authentication and the browser must prompt the user?
 * What happens when a proxy implements QUIC?

All of these are handled somewhere in the stack, but I've skipped over them to try to reduce the breadth of knowledge I had to understand. And it was still quite broad!