Chromium Spelunking: Churl
Tags: chromium, cpp
Published: 2023-05-08 18:47:41

At the end of my [last post](https://dev.to/djmitche/chromium-spelunking-life-and-times-536b) I indicated I would try to build a simple executable to fetch a URL, similar to curl. `churl` sounds like a suitably silly name for it.

This post is going to be a little on the long side -- it's leaning into the "lightly edited lab notebook" format, in hopes of highlighting some of the places I got stuck.

## An Executable

I want to focus on my goals here -- understanding the network stack, specifically proxies and QUIC -- so I don't want to get distracted with things like how to set up Ninja to create a new executable. So, I went looking for something that was near to what I wanted and which I could copy/paste and modify. I got lucky! There is a [`quic_client`](https://www.chromium.org/quic/playing-with-quic/) executable that seems _quite_ close, really. However, it's specific to QUIC and I want to make something that begins at `URLRequest`. Still, this gives me an easy way to create a new executable in `net/BUILD.gn` and do a simple "Hello World":

```c++
int main(int argc, char* argv[]) {
  DLOG(ERROR) << "Hello, world.";                                                                                                                                                                                  
}
```

## Getting to URLRequestContext

From the previous post, step one is going to be getting a `URLRequestContext` up and running, and that requires [a builder](https://source.chromium.org/chromium/chromium/src/+/main:net/url_request/url_request_context_builder.h).

### ThreadPoolInstance

Naively trying to create such a thing gets me an error because I haven't initialized a thread pool. The network stack does seem to use a lot of the task-posting magic, so I suppose that's necessary! A bit of guessing based on `base/test/task_environment.cc` leads me to

```c++
base::ThreadPoolInstance::Set(std::make_unique<base::internal::ThreadPoolImpl>("my histogram"));
```

and on to the next error. I am hoping to only have to make two or three of these "wild guesses" before things start working. Beyond that point, when something doesn't work it will be almost impossible for me to figure out which of the _n>3_ things I do not understand might be malfunctioning.

### SequencedTaskRunner

Next up, I need a `SequencedTaskRunner` to run tasks on that thread pool:
```
[0508/164407.189292:FATAL:post_task_and_reply_impl.cc(158)] Check failed: has_sequenced_context || !post_task_success. 
```

Checking [`sequenced_task_runner.h`](https://source.chromium.org/chromium/chromium/src/+/main:base/task/sequenced_task_runner.h?q=SequencedTaskRunner) shows a lot of docs _about_ the type, but not how to construct it. The bottom of the doc comments mentions some "theoretical implementations", suggesting that this is an abstract base class, and in fact a few methods are `= 0`. So, where are the implementations? Ordinarily I'd use CodeSearch for this, but it seems that the 91 subclasses are more than its little brain can handle.

I continued on a little down this path, but I am learning that when I find myself reading the Chromium source, I'm looking in the wrong place. In fact, the nice [Threading and Tasks](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/threading_and_tasks.md) document includes snippets that create task runners:
```c++
auto thread_runner = base::ThreadPool::CreateTaskRunner({base::TaskPriority::USER_VISIBLE});
```
The same error occurs, I think because while this creates a new task runner, it does not set it as the default. It looks like `CreateDefaultHandle` is an RAII wrapper for setting a task runner as the default, so let's try
```c++
base::SequencedTaskRunner::CurrentDefaultHandle active_thread_pool(thread_runner);
```

This call itself fails:
```
[0508/171849.025900:FATAL:sequenced_task_runner.cc(99)] Check failed: task_runner_->RunsTasksInCurrentSequence(). 
```
The comment docs for `RunsTasksInCurrentSequence` suggest that it is meant to be called within a task, so maybe I need to run the remainder of `churl` in a task in this runner? 
```c++
void Task() {
  net::URLRequestContextBuilder url_request_context_builder;
  url_request_context_builder.set_user_agent("Dustin's Experiment");

  auto url_request_context = url_request_context_builder.Build();

  DLOG(ERROR) << "Hello, world." << url_request_context;
}

int main(int argc, char* argv[]) {
  base::ThreadPoolInstance::Set(std::make_unique<base::internal::ThreadPoolImpl>("my histogram"));

  auto thread_runner = base::ThreadPool::CreateSequencedTaskRunner({base::TaskPriority::USER_VISIBLE});
  thread_runner->PostTask(FROM_HERE, base::BindOnce(Task));
}
```

Indeed, that compiles, but since `main` exits immediately after posting the task, it doesn't print "Hello, world."

Referring to the _Threading and Tasks_ document again, I tried `base::RunLoop().RunUntilIdle()`, but this seems to require that it be called with the current loop set, and that seems a bit circular. Lower down I see `base::RunLoop::Run`, which will call until a `QuitClosure` is run. That sounds better, but ultimately has the same issue:
```
[0508/173608.004631:FATAL:single_thread_task_runner.cc(44)] Check failed: handle. Error: This caller requires a single-threaded context (i.e. the current task needs to run from a SingleThreadTaskRunner). If you're in a test refer to //docs/threading_and_tasks_testing.md.
```
Further down the document:
```c++
// To block until all tasks posted to thread pool are done running:
base::ThreadPoolInstance::Get()->FlushForTesting();
```
But that just hangs without actually running the task. Even _further_ down the document, there's a section entitled "Using ThreadPool in a New Process", and it turns out that this is what I'm trying to do. It uses
```c++
base::ThreadPoolInstance::CreateAndStartWithDefaultParams("process_name");
```
instead of the initialization I was using. I don't really understand the difference, but this appears to make things work and gets the next error from `net::URLRequestContextBuilder::Build`:
```
[0508/174429.973862:FATAL:configured_proxy_resolution_service.cc(800)] Check failed: proxy_config_service. 
```

### Proxy Config Service / Proxy Resolution Service

Proxies! At least that's relevant to my interests.

Following the traceback for that failure:
```
#6 0x7f00db30f76f logging::CheckError::~CheckError()
#7 0x7f00dd28f8f7 net::ConfiguredProxyResolutionService::CreateUsingSystemProxyResolver()
#8 0x7f00dd63b1b7 net::URLRequestContextBuilder::CreateProxyResolutionService()
#9 0x7f00dd637485 net::URLRequestContextBuilder::Build()
```
it looks like `Build` is passing a NULL unique_ptr for `proxy_config_service` to `CreateProxyResolutionService`, and adding some debug prints confirms that. In fact, the debug prints confirm that `BUILDFLAG(IS_LINUX)` is not true, despite building this on a Linux system. I don't know a quick way to fix that, so hopefully I can work around it.

Commenting out the build conditionals in [`url_request_context_builder.cc`](https://source.chromium.org/chromium/chromium/src/+/main:net/url_request/url_request_context_builder.cc;l=445?q=net%2Furl_request%2Furl_request_context_builder.cc) gets a different error, I think to do with `ProxyConfigService::CreateSystemProxyConfigService` requiring a `SingleThreadTaskRunner` when we're using a `SequencedTaskRunner`, so maybe that's a dead-end.

Maybe I can provide a stubbed-out implementation of these types? It looks like `ProxyConfigService` can create a `ProxyResolutionService`, or I can just specify a `ProxyResolutionService` directly. Looking for subclasses of both abstract base classes, I found a local [`ProxyConfigServiceDirect`](https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:net/proxy_resolution/proxy_config_service.cc;l=79;drc=e0e0d24aaa54727dc0a8bc4b159ccdf80d3f5d8d;bpv=1;bpt=1) which just always returns DIRECT. That looks useful, so I'll copy-paste it (and add the necessary namespaces):
```c++
// Config getter that always returns direct settings.
class ProxyConfigServiceDirect : public net::ProxyConfigService {
 public:
  // ProxyConfigService implementation:
  void AddObserver(Observer* observer) override {}
  void RemoveObserver(Observer* observer) override {}
  net::ProxyConfigService::ConfigAvailability GetLatestProxyConfig(
      net::ProxyConfigWithAnnotation* config) override {
    *config = net::ProxyConfigWithAnnotation::CreateDirect();
    return CONFIG_VALID;
  }
};

void Task() {
  net::URLRequestContextBuilder url_request_context_builder;
  url_request_context_builder.set_proxy_config_service(
    std::make_unique<ProxyConfigServiceDirect>());
  // ..
}
```
and with that, I've successfully built a `UrlRequestContext`!

## Checking In

OK, so I've made some progress today. A lot of this has been trial-and-error. There are two problems with this approach. First, it means I don't really learn how things work -- for example, I don't know why so many options for setting up a task runner failed. But I can add an item to my task list to learn more about that some day, and carry on with my current project.

Second, the distinction between "OK" and "Error" in this kind of trial-and-error is not always clear. In this case, I thought I had finished the creation of the thread pool successfully, but it turned out I had to revisit this and do it a different way. That was down to luck, and with a few more repetitions of trial-and-error, the universe of unknown things that could be wrong is so large that no amount of luck is enough to make progress.

The tools also let me down a bit here: CodeSearch was overwhelmed and unable to give me answers. I omitted it above, but I fell back to `git grep`. Like a pocket-knife, it's a tool that isn't fancy but is always there and always works. Another go-to tool that I used here was debug prints (in this case, `DLOG(ERROR) << "HERE 1";` and incrementing the number). It's not especially cool, but it provides reliable, ground-truth information.

The code itself let me down a little, too. Classes in `base` seem to be well-commented, but with deep technical detail and not answers to questions like "how do you create an instance". Maybe that's OK -- it's a lot to ask from code comments -- but it's a reminder to me that I need to look outside of the code for higher-level documentation.