Using mitmproxy with k6
Tags: performance, programming, network, beginners
Published: 2023-12-09 10:35:01


Mitmproxy is one of the best-known proxies used in everyday work. What sets it apart from Fiddler and Charlesproxy is that it is open source.

The tool provides us with three main modes of operation. Each of them has a different task, and their use will depend on what we want to achieve.

1. `mitmproxy` - an interactive, SSL/TLS-enabled proxy with a console interface for HTTP/1, HTTP/2 and WebSockets. Its main task is to track passing traffic.
2. `mitmdump` - a command line companion to mitmproxy. It provides functionality similar to tcpdump, allowing you to view, log and programmatically transform HTTP traffic. In other words, mitmdump is used to record, filter and aggregate network traffic.
3. `mitmweb` - a web interface for mitmproxy covering the point of functionality described in point 1 and 2.

## Why do we need mitmproxy in k6?

One of the fundamental problems in the context of writing performance tests is capturing traffic from both mobile and web applications.

Mitmproxy's target task is to record network traffic into a suitable format, which can then be quickly transferred to a k6 scenario and then parameterized.

## Installation

Installation, depending on the operating system, is done in several ways. The simplest is to use `pip3` packet manager.

```bash
pip3 install mitmproxy
```

## mitmproxy in action

The requests and responses intercepted by mitmproxy are called `HTTP flows`. While they can be easily converted, this is not the `HAR` format. Let's check step by step how.

### Capture requests and responses

A simple startup of mitmproxy in console mode is done by the command:

```bash
mitmproxy
```

After running the above command, an interactive console will open. For the time being, no traffic is visible in it, as nothing has been sent through the proxy yet.

After configuring, for example, JMeter (in `HTTP Request Defaults` set as `Server IP` the address `127.0.0.1` and the port as `8080` [default]) or a browser to use the proxy, records will begin to appear in the console.

```bash
Flows
21:07:08 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       342ms
21:07:09 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       209ms
21:07:09 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       241ms
21:07:10 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       270ms
21:07:10 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       240ms  
```

These are the `HTTP Flows` that we mentioned earlier.

### Save `HTTP Flows`.

After generating the appropriate traffic in the console, we press the `w` button to go to save `HTTP Flows`, and then type the path to the file to be created.

```powershell
: save.file @shown C:\Users\BackBugs\Desktop\flows
```

This will save a file named `flows` on the desktop. Unfortunately, it doesn't accomplish much in our case.

To get a file in `.HAR` format, which we can then convert to k6 script, we can't use the saved `HTTP flows`. This is because converting them from file level to `HAR` format is impossible. However, it is possible to save real-time traffic captured by `mitmproxy` to `HAR` format. This is done by `mitmdump`.

### Converting `HTTP Flows` to `HAR`.

Files in `HAR` format are standard in all web browsers. In the `mitmproxy` repository, we can find [script]([https://github.com/mitmproxy/mitmproxy/blob/main/examples/contrib/har_dump.py](https://github.com/mitmproxy/mitmproxy/blob/main/examples/contrib/har_dump.py)) with which it is possible to kownersion `HTTP Flows` to `HAR` archives. It is done through the following command.

```
mitmdump -s ./examples/contrib/har_dump.py --set hardump=./dump.har
```

The console will show summaries of the requests.

```bash
[11:40:41.279] Loading script /har-dump.py
[11:40:41.284] HTTP(S) proxy listening at *:8080.
[11:40:45.051][127.0.0.1:12464] client connect
[11:40:45.142][127.0.0.1:12464] server connect k6.io:443 (18.66.233.97:443)
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
[11:40:47.201][127.0.0.1:12464] client disconnect
[11:40:47.202][127.0.0.1:12464] server disconnect k6.io:443 (18.66.233.97:443)
[11:40:48.702] HAR dump finished (wrote 2672545 bytes to file)
```

## HAR conversion to k6 script.

The next step is already the target conversion of the `HAR` format file to k6 script. For this we use [script](https://github.com/grafana/har-to-k6) provided in Grafana's repository.

```bash
har-to-k6 dump.har -o k6-scenario.js
```

After installing and using the above command, a test scenario will be generated. Unfortunately, har-to-k6 is not an ideal tool. This is not due to its primitiveness, but to its inability to cover, for example, dynamic paths in some of the applications. Therefore, such elements as parameterization or `checks` we have to add ourselves.

An example of how the script looks after conversion:

```javascript
/*
 * Creator: mitmproxy har_dump 0.1
 * mitmproxy version mitmproxy 9.0.1
*/

import { sleep } from 'k6'
import http from 'k6/http'


export const options = {}
  
export default function main() {
  let response
  
  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })
  
  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })
  
  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })
  
  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })
  
  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })

  // Automatically added sleep
  sleep(1)
}
```