AREX: How to Debug API in the browser
Tags: frontend, programming, opensource
Published: 2023-10-12 07:00:59

<!-- TOC start (generated with https://github.com/derlin/bitdowntoc) -->

- [What is AREX?](#what-is-arex)
- [Challenge 1: Cross-Origin Restrictions](#challenge-1-cross-origin-restrictions)
   * [Solutions](#solutions)
- [Challenge 2: API Debugging](#challenge-2-api-debugging)
   * [Solutions](#solutions-1)
- [Challenge 3: Binary serialization](#challenge-3-binary-serialization)
   * [Solution](#solution)
   * [Code Analysis](#code-analysis)
      + [File to Base64](#file-to-base64)
      + [Base64 to File](#base64-to-file)

<!-- TOC end -->

Debugging APIs is vital in software development. It is a critical step to validate and test the effectiveness and correctness of application interfaces. However, traditional API debugging methods typically rely on separate tools or desktop applications, limiting the debugging process's flexibility and efficiency.

This article will use the open-source project AREX as an example to introduce how to implement API debugging functionality in the browser.

## What is AREX?

[AREX](http://arextest.com/) is an open-source automated regression testing platform based on real requests and data. It utilizes Java Agent technology and comparison techniques to achieve fast and effective regression testing through traffic recording and playback capabilities.

## Challenge 1: Cross-Origin Restrictions

To debug API in the browser, the first challenge to overcome is handling the browser's cross-origin restrictions.

The browser cross-origin issue refers to the restrictions encountered when using JavaScript code to access resources from one domain's web pages to another domain's resources in web development. Browsers implement a security policy called the Same-Origin Policy to protect user information. This policy is **a browser security feature that restricts how documents and scripts on one origin can interact with resources on another origin**.

Due to the browser's security policy, HTTP requests cannot be sent from within the browser due to cross-origin restrictions.

### Solutions

There are two common solutions to overcome this limitation: Chrome extension proxy and server-side proxy. Now we compare the two methods:

|                | Chrome extension                                             | Server-side                                                  |
| -------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
| local access   | √                                                            | ❌                                                            |
| speed          | No request time loss                                         | The overall process speed is influenced by the proxy interface. |
| Actual request | The request `Origin` will be modified to match the source of the Chrome extension | the same as the actual request                                                    |

After weighing the options, AREX chose Chrome extension proxy. 

The background of the Chrome extension has the ability to send cross-domain requests. Then we can intercept the requests received on the browser side and communicate with the background of the Chrome extension using `window.postMessage` (where communication also requires the `content-script` of a Chrome extension to serve as a data bridge).

The implementation is as follows:

- In the client-side script:

1. Generate a random string and store it as a string in the variable `tid`.
2. Use the `window.postMessage()` method to send a message to other extensions. The message includes an identifier of type `AREX_EXTENSION_REQUEST`, `tid`, and the `params` parameter.
3. Add a `message` event listener `receiveMessage` to receive messages sent by other extensions.
4. In the `receiveMessage` function, check if the received message is of type `AREX_EXTENSION_RES` and if the `tid` matches the one sent in the previous message. If there is a match, remove the event listener.

- In the content script:

1. Add a `message` event listener to receive messages sent from the page script or other extension programs. 
2. Within the event listener, verify if the received message is of type `AREX_EXTENSION_REQUEST`. If so, utilize the `chrome.runtime.sendMessage()` method to dispatch the message to the background script.
3. After receiving a response from the background script, use the `window.postMessage()` method to send the response message back to the page script or other extension programs.

- In the background script:

1. Use the `chrome.runtime.onMessage.addListener()` method to add a listener for receiving messages from content scripts or other extension programs. 
2. Within the listener, you can handle the received messages and respond accordingly based on your requirements.

```javascript
// arex
const tid = String(Math.random());
window.postMessage(
  {
    type: '__AREX_EXTENSION_REQUEST__',
    tid: tid,
    payload: params,
  },
  '*',
);
window.addEventListener('message', receiveMessage);
function receiveMessage(ev: any) {
  if (ev.data.type === '__AREX_EXTENSION_RES__' && ev.data.tid == tid) {
    window.removeEventListener('message', receiveMessage, false);
  }
}
// content-script.js
window.addEventListener("message", (ev) => {
  if (ev.data.type === "__AREX_EXTENSION_REQUEST__"){
    chrome.runtime.sendMessage(ev.data, res => {
      //  communicate with background
      window.postMessage(
        {
          type: "__AREX_EXTENSION_RES__",
          res,
          tid:ev.data.tid
        },
        "*"
      )
    })
  }
})
// background.js
chrome.runtime.onMessage.addListener((req, sender, sendResponse) => {

})

```

## Challenge 2: API Debugging

The above has resolved the cross-origin restrictions. The next challenge is how to implement the API debugging feature in the browser.

### Solutions

Postman is a mature API debugging tool. By standing on the shoulders of giant, we introduced the JavaScript sandbox of Postman in AREX. We utilize the sandbox to run pre-request scripts, post-request scripts, and assertions for API debugging.

Here is the flowchart of request in AREX :

![request flowchart](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fcz9s6vag0budaklf03k.png)

When sending the request, the data in the form will be gathered together into a data structure as below:

```css
export interface Request {
  id: string;
  name: string;
  method: string;
  endpoint: string;
  params: {key:string,value:string}[];
  headers: {key:string,value:string}[];
  preRequestScript: string;
  testScript: string;
  body: {contentType:string,body:string};
}
```

This is the data structure of AREX, which will be converted into Postman's. Then, invoke the PostmanRuntime.Runner() method, and pass the converted Postman data structure and the currently selected environment variables. The Runner will execute the `preRequestScript` and `testScript` . The `preRequestScript executes *before* a *request* runs` and can include other requests as well as operations on request parameters and environment variables. The `testScript` is after the request and can perform assertions on the response. Additionally, the scripts can use `console.log` to output data for debugging in the console.

```
var runner = new runtime.Runner(); // runtime = require('postman-runtime');

// A standard Postman collection object
var collection = new sdk.Collection();

runner.run(collection, {}, function (err, run) {
    run.start({
      assertion:function (){},
      prerequest:function (){}, 
      test:function (){}, 
      response:function (){} 
    });
});
```

The cross-origin issues also exist in the Postman sandbox. Due to the high integration of the Postman sandbox,  we have adopted Ajax (Asynchronous JavaScript and XML) interception technique to ensure convenience and synchronization with `PostmanRuntime`. By intercepting Ajax requests on the browser side, we can modify requests, add custom logic, or perform other processing operations.

When the Postman sandbox sends a request, it carries a request header "postman-token". After intercepting Ajax's request, we assemble the request parameters and send them to the browser extension via `window.postMessage`. The browser extension then constructs a fetch request and returns the data to the Postman sandbox. Then Postman sandbox outputs the final results, including the response, test results, and `console.log`. The responseType should be specified as "arraybuffer".

The specific process:

1. Register a request handler using the `xspy.onRequest()` method. This handler accepts two parameters: `request` and `sendResponse`. The `request` parameter contains relevant information about the request, such as the method, URL, headers, request body, etc. `sendResponse` is a callback function used to send a response back to the requester. 
2. In the handler, check if there is `postman-token` in the request headers to determine if the request is from Postman.

- If the request is from Postman, use `AgentAxios` to send a new request with the same details. The response is stored in `agentData`. Create a `dummyResponse` object with relevant information from the original request. Set the status, headers, ajaxType, responseType, and response fields of `dummyResponse` based on `agentData`. Finally, return the dummyResponse using `sendResponse()`.

- If not, then call `sendResponse()` directly, indicating that no response should be returned.

```typescript
xspy.onRequest(async (request: any, sendResponse: any) => {
  // check if is sent from postman
  if (request.headers['postman-token']) {
    const agentData: any = await AgentAxios({
      method: request.method,
      url: request.url,
      headers: request.headers,
      data: request.body,
    });
    const dummyResponse = {
      status: agentData.status,
      headers: agentData.headers.reduce((p: any, c: { key: any; value: any }) => {
        return {
          ...p,
          [c.key]: c.value,
        };
      }, {}),
      ajaxType: 'xhr',
      responseType: 'arraybuffer',
      response: new Buffer(JSON.stringify(agentData.data)),
    };
    sendResponse(dummyResponse);
  } else {
    sendResponse();
  }
});
```

## Challenge 3: Binary serialization

For requests with data formats of `x-www-form-urlencoded` and `Raw`, handling them is relatively easy since they are regular JSON objects. However, for requests of `form-data` and `binary`, they need support for sending binary payload. Unfortunately, the `postMessage` communication method in Chrome extensions does not support directly passing binary files, which makes it impossible to handle these two types of requests directly.

### Solution

To address this issue, AREX utilizes the base64 encoding technique. When a user selects a file, AREX converts the binary file into a base64 string for transmission. On the Chrome extension side, AREX decodes the base64 data and uses it to construct the actual `fetch` request. This approach bypasses the limitation of directly passing binary files.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0h1pe852ra9e8exhyxih.png)

This flowchart describes the process of converting binary files in FormData to Base64 strings, and then converting them back to files and further processing them through the Chrome proxy extension.

1. form-data binary: a FormData contains a binary file. 
2. FileReader: Uses a FileReader object to read the binary file. 
3. readAsDataURL base64 string: The FileReader uses the `readAsDataURL` method to read the binary file as a Base64 string. 
4. Chrome extension proxy: After the Base64 string is read, it is passed to the Chrome proxy extension. 
5. base64 string: Represents the Base64 string obtained after the binary file is read by the FileReader. 
6. Uint8Array: In the Chrome extension proxy, the Base64 string is converted to a Uint8Array. 
7. File: A new File object is created using the data from the Uint8Array. 
8. fetch: The new File object is further processed using the fetch method or other means, such as uploading to a server or performing other operations.

### Code Analysis

Now let's look at the code level:

```ini
// File to Base64
const toBase64 = (file: File): Promise<string> =>
  new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.readAsDataURL(file);
    reader.onload = () => resolve(reader.result as string);
    reader.onerror = reject;
  });
// Base64 to File
function base64ToFile(dataurl: string, filename: string) {
  const arr = dataurl.split(',') || [''],
    mime = arr[0].match(/:(.*?);/)?.[1],
    bstr = atob(arr[1]);
  let n = bstr.length;
  const u8arr = new Uint8Array(n);
  while (n--) {
    u8arr[n] = bstr.charCodeAt(n);
  }
  return new File([u8arr], filename, { type: mime });
}
​
export default base64ToFile;
```

#### File to Base64

The `toBase64` function takes a `File` object as input and returns a `Promise` that resolves to a Base64 string representing the file.

Inside the function, a `FileReader` object is used to read the contents of the `File` object as a Data URL by calling `reader.readAsDataURL(file)`.

The `onload` event handler is set on the `FileReader` object, which will be triggered when the file reading operation is complete. Inside the event handler, the `reader.result` property, which contains the result of the file reading operation, is resolved as a `string` and passed to the `resolve` function of the `Promise`.

If an error occurs during the file reading operation, the `onerror` event handler is triggered, and the `reject` function of the `Promise` is called with the error.

#### Base64 to File

`The function base64ToFile` takes two parameters: `dataurl` (a Base64 string) and `filename` (the name of the file). The function returns a `File` object that represents the file with the specified name and content.

Inside the function, the Base64 string is split into an array using the comma separator. The first element of the array contains the MIME type of the file, which is extracted using a regular expression. The second element of the array contains the Base64-encoded content of the file, which is decoded using the `atob` function.

The decoded content is then converted to a `Uint8Array` object, which is used to create a new `File` object. The `File` object is returned with the specified filename and MIME type.

The above is the internal mechanism of the entire AREX debugging process.

---

Community⤵️
🐦 Follow us on [Twitter](https://twitter.com/AREX_Test)
📝 Join AREX [Slack](https://arexcommunity.slack.com/ssb/redirect)
📧 Join the [Mailing List](https://groups.google.com/g/arex-test)