Gathering HAR and deciphered PCAP from you webdriver tests
Tags: dotnet, selenium, mitm, csharp
Published: 2024-01-15 18:20:19

To gather network data without relying solely on the browser, we need an intermediary that can monitor the traffic between the browser and the remote server. For the following tutorial, we will be using [Fluxzy](https://github.com/haga-rak/fluxzy.core), an open-source Man-In-The-Middle (MITM) intermediary that can be used as a .NET library. We will use it along with Selenium to extract HTTP Archive (HAR) Files and PCAPNG from any browser or remote devices.

## The code 

Let's start by creating a new console project. For all the following, I'm using .NET 8.

```bash
dotnet new console -o webdriver_with_har
cd webdriver_with_har
```

Install [fluxzy packages](https://www.nuget.org/packages/Fluxzy.Core.Pcap/):

```bash
dotnet add package Fluxzy.Core.Pcap
```

With a quick check on the main docs, we can run the proxy with a few lines of code

```csharp

using Fluxzy;

var fluxzySetting = FluxzySetting.CreateDefault()
    .SetOutDirectory("temp_directory");

await using (var proxy = new Proxy(fluxzySetting))
{
    _ = proxy.Run(); 
    // The proxy now runs until this scope is disposed
}

Packager.ExportAsHttpArchive("temp_directory", "export.har");
```

The code above is quiet straightforward. We create a new proxy instance with the default settings.
Then, we start the proxy and wait for the proxy to be disposed. Once the proxy is disposed, we export the network dump into an HTTP Archive file.

For this example above will generate an empty HAR as we did not redirect actual traffic to the proxy.

## Integrate with Selenium

Let's add Selenium to the project:

```bash
dotnet add package Selenium.WebDriver
```

[`Proxy.Run()`](https://docs.fluxzy.io/api/Fluxzy.Proxy.html#Fluxzy_Proxy_Run) method returns all endpoints on which the proxy is listening. We can use any of them to configure the webdriver.

```csharp
var endpoint = proxy.Run().First(); 

var options = new FirefoxOptions
{
    Proxy = new OpenQA.Selenium.Proxy
    {
        HttpProxy = $"{endpoint.Address}:{endpoint.Port}",
        SslProxy = $"{endpoint.Address}:{endpoint.Port}"
    },
    // Necessary if you did not install the root certificate computer-wide
    AcceptInsecureCertificates = true 
};
```

Note that, if the root certificate is not installed computer-wide, you need to set `AcceptInsecureCertificates` to `true`. Otherwise the browser will trigger errors when connecting to the proxy.

Now, a full working example making a request to google.com and exporting the HAR file would look like this:

```csharp
using Fluxzy;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;

var fluxzySetting = FluxzySetting.CreateDefault()
    .SetOutDirectory("temp_directory");

await using (var proxy = new Fluxzy.Proxy(fluxzySetting))
{
    var endpoint = proxy.Run().First(); 
    var options = new FirefoxOptions
    {
        Proxy = new OpenQA.Selenium.Proxy
        {
            HttpProxy = $"{endpoint.Address}:{endpoint.Port}",
            SslProxy = $"{endpoint.Address}:{endpoint.Port}"
        },
        AcceptInsecureCertificates = true
    };

    using var driver = new FirefoxDriver(options); 

    driver.Navigate().GoToUrl("https://www.google.com");
    driver.FindElement(By.TagName("textarea")); 
    driver.Quit();
}

Packager.ExportAsHttpArchive("temp_directory", "export.har");
Packager.Export("temp_directory", "export.fxzy");

```

## Going further with PCAPNG data extraction

PCAPNG (stands for PCAP new generation) is a newer version of the PCAP format. It fullfills basically the same purpose as regular PCAP and can be open with a recent version of Wireshark. What's interesting with the fluxzy integration is its ability to hold, out of the box, NSS Key log datas which can be used to view the decrypted TLS traffic in Wireshark.

To enable NSS Key log datas, we instruct fluxzy to use the Bouncy Castle library during the proxy setup:

```csharp
fluxzySetting.UseBouncyCastleSslEngine();
```

Then we can configure the proxy to enable PCAPNG extraction.

```csharp
await using (var tcpConnectionProvider = await CapturedTcpConnectionProvider.CreateInProcessCapture())
await using (var proxy = new Fluxzy.Proxy(fluxzySetting, tcpConnectionProvider: tcpConnectionProvider)){
    // Use proxy
}
```

The code above tells fluxzy to use a `tcpConnectionProvider` that will capture any created TCP connections into a PCAPNG file.


## Full code with HAR and pcapng 

Here is a full working example with HAR and PCAPNG extraction: 

```csharp
using System.Net;
using Fluxzy;
using Fluxzy.Core.Pcap;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;

var fluxzySetting = FluxzySetting.CreateDefault(IPAddress.Loopback, 0)
    .SetOutDirectory("temp_directory");

fluxzySetting.UseBouncyCastleSslEngine();

await using (var tcpConnectionProvider = await CapturedTcpConnectionProvider.CreateInProcessCapture())
await using (var proxy = new Fluxzy.Proxy(fluxzySetting, tcpConnectionProvider: tcpConnectionProvider))
{
    var endpoint = proxy.Run().First(); 
    var options = new FirefoxOptions
    {
        Proxy = new OpenQA.Selenium.Proxy
        {
            HttpProxy = $"{endpoint.Address}:{endpoint.Port}",
            SslProxy = $"{endpoint.Address}:{endpoint.Port}"
        },
        AcceptInsecureCertificates = true
    };

    using var driver = new FirefoxDriver(options); 

    driver.Navigate().GoToUrl("https://www.google.com");
    driver.FindElement(By.TagName("textarea")); 
    driver.Quit();
}

Packager.ExportAsHttpArchive("temp_directory", "export.har");
Packager.Export("temp_directory", "export.fxzy");
```

## Getting the PCAPNG file 

The PCAPNG file is not exported directly because it's sliced individually for each TCP connection made by the browser. 

There are, two non programmatic ways to get the PCAPNG file for a specific HTTP exchange or connection.

### Using Fluxzy CLI

[Fluxzy CLI](https://www.fluxzy.io/resources/cli/overview) is a command line tool that can be used to access most of fluxzy features without writing a single line of code.

The command that we are interested in is `fluxzy dissect`. It allows to make various read operations on a fluxzy dump.

First we list the content of the archive:

```bash
fluxzy dissect export.fxzy
```

This should output something like this:

```
...
22 - https://www.google.com/ - 200
23 - https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png - 200
24 - https://shavar.services.mozilla.com/downloads?client=navclient-auto-ffox&appver=121.0&pver=2.2 - 200
25 - https://www.google.com/gen_204?atyp=i&ct=bxjs&cad=&b=0&ei=2GulZf2oCbuikdUP6LCs8As&zx=1705339867346&opi=89978449 - 204
26 - https://www.google.com/ - 200
27 - https://www.google.com/gen_204?ei=smulZYG4GLqrkdUPyI6RGA&vet=10ahUKEwiB2cv09d-DAxW6VaQEHUhHBAMQhJAHCB8..s&bl=xaqk&s=webhp&gl=fr&pc=SEARCH_HOMEPAGE&isMobile=false - 204
...
```

Each line of the output is a specific HTTP exchange. The first column is the ID of the exchange. The second column is the URL of the exchange. The third column is the HTTP status code of the exchange. This output is customizable with the `--format` option but now, what we want is to get the id of the exchange that we are interested in.

To extract the PCAPNG matching the exchange with id 22, we can run the following command:

```bash
fluxzy dissect export.fxzy -i 22 -f "{pcap}" -o exchange22.pcapng
```

The `-i` option is used to specify the id of the exchange. The `-f` option is used to specify the format of the output. The `{pcap}` placeholder is used to specify that we want the PCAPNG file. The `-o` option is used to specify the output file which is `exchange22.pcapng` in this case.

### Using Fluxzy Desktop 

[Fluxzy Desktop](https://www.fluxzy.io/) is a free HTTP debuggger that provides a graphical way to access easily a fluxzy archive. To extract the PCAPNG file of an exchange, you can simply select the matching exchange and choose "Connection -> Pcapng -> Download". And that's all. 

![Accessing PCAPNG file with Fluxzy Desktop](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6c8y2qcfg74szye33dhe.png)

The downloaded PCAPNG file should be opened deciphered in wireshark without any extra handling.

![Deciphered PCAPNG file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t7vo4ohts35wilrzmgpt.png)

## Conclusion


That's it! We've demonstrated how to integrate Fluxzy with Selenium for effective network data extraction in web testing.  This approach not only simplifies the process of capturing HTTP and network traffic data but also opens up new possibilities for in-depth analysis and debugging.