Repository: UKGovernmentBEIS/inspect_ai
Issue: Add ability to use self-signed SSL certs with web browser tooling
Root cause: Unimplemented Feature Gap

For some evals, it is useful to be able to mock a website that the agent already knows about. For example, if you ask the agent to access Facebook, it will try and access `https://www.facebook.com`. It is very useful to have that resolve to a mock web server transparently to the agent.

By manipulating the Compose configuration, you can have `www.facebook.com` resolve to another container running a TLS HTTP server. However, because Chromium uses its own NSS certificate store database, modifying the system certs won't prevent the agent seeing errors when it navigates to mocked pages.

There are a few workarounds which I'm exploring here:
1. In Playwright's Chromium browser `new_context` method, we can pass the flag `ignore_https_errors=True`, which will ignore all HTTPS errors. This is the least preferable option because some HTTPS errors are worth respecting.
2. Similarly, we can pass the flag `client_certificates` to the `new_context` method, which looks like it would allow us to trust particular certificates. However, this is kind of difficult from an implementation perspective.
3. Most preferably, the user could just alter the Chromium certificate store from the Dockerfile before starting the web server so that the certificates are trusted without any modifications to the web browser base image or any additional configuration.

I'm going to research option 3 to see if that is possible. Otherwise, I think it's worth doing a temporary hack by adding a CLI flag to ignore HTTPS errors while proper configuration for client certificates is investigated.


I've gotten 80% of the way there, where I can generate certificates that the web browser container OS trusts, and I can get them trusted by the NSSDB that Chromium is supposed to use, but for whatever reason I'm still getting blank responses in the eval. I suspect this has to do with where the NSSDB that Playwright Chromium actually uses is located. 

Will try and hack on this for a bit more, otherwise we might want to proceed with fallbacks. 

Alright, I give up on the clean approach. Likely Playwright Chromium is bundled up in a way that doesn't read the NSSDB or otherwise can't verify the certificates in the way I am using them. I suggest we now just add a flag to the web server such that it ignores certificate errors.

For reference, these are the steps that I took:

```bash
# Generate root CA private key
#openssl genrsa -out rootCA.key 4096
# Generate root CA certificate
#openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt
# Generate server private key and CSR
openssl req -new -sha256 -key facebook.com.key -subj "/C=US/ST=CA/O=MyOrg, Inc./CN=facebook.com" -out facebook.com.csr
# Verify the CSR
openssl req -in facebook.com.csr -noout -text
# Generate server certificate using the root CA
openssl x509 -req -in facebook.com.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out facebook.com.crt -days 500 -sha256
# Verify the server certificate
openssl x509 -in facebook.com.crt -text -noout
```

The browser Dockerfile I tried to use was something like:
```
FROM aisiuk/inspect-web-browser-tool

WORKDIR /app/web_browser

COPY . .
RUN apt install libnss3-tools

RUN cp rootCA.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates --fresh

RUN mkdir -p $HOME/.pki/nssdb
RUN certutil --empty-password -d $HOME/.pki/nssdb -N
RUN certutil -d sql:$HOME/.pki/nssdb -A -t "CT,," -n rootCA.crt -i /usr/local/share/ca-certificates/rootCA.crt

CMD ["python3", "/app/web_browser/web_server.py"]
```

I then had a web server serving the `facebook.com` cert with the appropriate aliases set up to resolve properly. `curl` accepted the certificate, but Playwright Chromium apparently did not.

resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/773