Replaying network requests on puppeteer
Tags: puppeteer, javascript, headless, hacking
Published: 2022-11-28 10:04:26

You possibly tried to read the request header or response body of incoming and outgoing requests by network intercepting if you’ve made some developments on puppeteer headless browser scripts. I’ve been working on puppeteer scripts for reverse engineering and scraping purposes and realized that there is no request replaying method. I needed that ― so I made a plugin for that: [https://www.npmjs.com/package/puppeteer-extra-plugin-replayer](puppeteer-extra-plugin-replayer)

## How to replay requests?

First of all, I want to explain how it works. CDP and Devtools allow you to copy a request with its body and headers as a prepared code by using the “Copy as Fetch code” option. If you want to replay the request, it’s enough to paste it into the JavaScript console and modify the requested content if it’s needed and send it.

_Voilà_, you replayed a request. It’s that easy.

>  But, what about CORS policies? No worries about that, since you sent that code through the console ― so the requests fit into the CORS policies. If you send the request through your bash CLI as a curl command or on another tab console, your request could be rejected ― if we assume that there are proper CORS policies on the server.

![The diagram shows how the script works](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xwalbxe3ciyb3dyw0ue9.png)

## The methods

There are two methods:
- `.catchRequest` in the `page` object
- `.replay` method that in the return value of `.catchRequest`

You can check the [interface declarations](https://github.com/sametcodes/puppeteer-extra-plugin-replayer/blob/main/dist/index.d.ts) to see the exact values.

`.catchRequest` takes two arguments. The first is a regex pattern to catch requests by their URLs, and the second is a callback method to trigger the wanted request. If it’s being triggered initially as the page opens, then `page.goto()` must be called in the trigger callback function.

```javascript
(async () => {
  const request = await page.catchRequest({ pattern: /task.json\?taskname=login/},
		() => page.goto("https://twitter.com"));
  const response = await request.replay();
})();
```

If the request is being triggered as clicks on a button, then `.click()` function must be defined in the callback as here:

```javascript
(async () => {
		await page.goto("https://twitter.com");

    const request = await page.catchRequest({pattern: /task.json\?taskname=login/},
			async () => {
				await page.waitForSelector("a[href='/login']")
        await page.click("a[href='/login']")
			});

    const response = await request.replay();
})();
```

`.replay()` can take `url`, `method`, `headers`, and `body` as optional parameters, and you can modify the request by redefining them when replaying.

```javascript
const response = await request.replay({
	url: "https://twitter.com/logout", // defining a new URL is possible
	method: "POST", // changing the request method is possible as well
 	body: JSON.stringify({test: true}),
	headers: {test: true},
});
```

If you want to get and use the default values of the request instead of overwriting, use a callback function and return the new value.

```javascript
const response = await request.replay({
	url: url => url.replace(/login/, "/logout"),
	method: "POST", // changing the request method is possible as well
 	body: body => body + ";test=true",
	headers: headers => {...headers, test: true}
});
```

### One last note

You can check out the CDP Network tab if you want to see replayed requests during the development process. Notice that the initiator value of the requests is different.


![How the replayed request seems on the network tab](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qnkb6uqaoypxmktm2a60.png)

## Conclusion

The package helps to replay any XHR requests, but it' possible to catch and replay any requests.

If you want to see the source code and contribute to the project, please check the [repository](https://github.com/sametcodes/puppeteer-extra-plugin-replayer).

_Happy coding!_