How to reverse engineer a JSON API on a single page application
Tags: webscraping
Published: 2022-12-26 13:13:19

---
canonical_url: https://serpapi.com/blog/how-to-reverse-engineer-a-json-api-spa/
description: Mitmproxy can be used to find the HTTP request with the needed data when browser developer tools can't find it
---

Websites like Bing Image Search and Walmart render pages with JavaScript and deliver page content via JSON APIs. While it's possible to [scrape dynamic web pages using the browser automation](https://serpapi.com/blog/puppeteer-antipatterns/#using-puppeteer-when-other-tools-are-more-appropriate), I prefer fetching data from the API endpoints directly. It *usually* (not always) works faster and more reliable.

I was debugging the Bing Image Search to help implementing our new [Bing Reverse Image Search API](https://github.com/serpapi/public-roadmap/issues/399). Initially, I've used [`mitmproxy`][1] because [`Ctrl+Shift+F` in the browser dev tools](https://developer.chrome.com/docs/devtools/search/#search-loaded-resources) haven't found the request. Then I've figured out how to filter network requests in the browser dev tools, examined the response, and made a draft data adapter.

## Algorithms to reverse engineer a JSON API on the SPA

Two ways I've used to reverse engineer a JSON API used on the Bing Image Search: `mitmproxy` and browser developer tools. I explain the devtools process because it's used more often.

### Browser devtools

1. `Ctrl+F` in the Network tab of browser dev tools.

  ![image](https://user-images.githubusercontent.com/282605/209423751-14a7a2a1-c8c9-43af-951c-4304648b02eb.png)

2. Go to the `Preview` tab of the JSON response.
3. Expand JS object recursively (my Brave Browser doesn't search in the collapsed JSON 😕)

  ![image](https://user-images.githubusercontent.com/282605/209423926-a0f2fd3d-8f58-4106-9432-2418e1233125.png)

5. `Ctrl+F` the target string

  ![image](https://user-images.githubusercontent.com/282605/209424004-7bbbf0b5-6aeb-43c4-9bee-a7bb05cb24b7.png)

6. Copy property path

  ![image](https://user-images.githubusercontent.com/282605/209423862-bf8a78b5-45bb-4a09-8e1b-b63b176e99f2.png)

7. Navigate up and down in JS object (with arrow keys) to learn its structure and create an adapter.
8. Copy as cURL and transform response with `jq` to check my assumption.

### `mitmproxy`

[`Ctrl+Shift+F` in the browser dev tools](https://developer.chrome.com/docs/devtools/search/#search-loaded-resources) no longer searches across all responses.

![image](https://user-images.githubusercontent.com/282605/209424769-a5b19cbe-832d-49a5-8b5f-abe1dd2fbc66.png)

I've proxied the browser network connections via [`mitmproxy`][1]. Then [filtered](https://docs.mitmproxy.org/stable/concepts-filters/) response bodies with `~bs "TEXT_FROM_THE_HTML_ELEMENT_I_"LOOKING_FOR"`.


1.  Start `mitmproxy` with view filter

```bash
$ mitmproxy --view-filter '~bs "Freshsales"'
```

2. Start chromium-based browser with the target URL and the following flags and parameters

* Proxy requests via `mitmproxy`: `--proxy-server='http://127.0.0.1:8080'`.
* Use incognito mode (1) with a temporary user profile (2) ignoring insecure connections (3) and certificate errors (4): `--temp-profile -incognito --user-data-dir="`mktemp -d`" --no-first-run --ignore-certificate-errors --allow-insecure-localhost`. (_I ignore certificate errors in a temporary browser profile to not install `mitmproxy`'s certificates system-wide._)

```bash
$ brave-browser 'https://www.bing.com/images/search?view=detailV2&insightstoken=bcid_RLKVsIV2BwkFXg*ccid_spWwhXYH&form=SBIHMP&iss=SBIUPLOADGET&sbisrc=ImgPicker&idpbck=1&sbifsz=927+x+524+%c2%b7+25.15+kB+%c2%b7+png&sbifnm=serpapi-serpbear.png&thw=927&thh=524&ptime=223&dlen=34344&expw=798&exph=451&selectedindex=0&id=-1051855017&ccid=spWwhXYH&vt=2&sim=11' --proxy-server='http://127.0.0.1:8080'  --temp-profile -incognito --user-data-dir="`mktemp -d`" --no-first-run --ignore-certificate-errors --allow-insecure-localhost
```

![image](https://user-images.githubusercontent.com/282605/209424552-595a65a4-5b89-43fd-8020-583fefb151fb.png)

3. `mitmproxy` will display the matched requests

![image](https://user-images.githubusercontent.com/282605/209424575-7516b36f-2883-4192-bcb9-c8e7c929e9e0.png)

## Conclusion

`mitmproxy` can be used to find the HTTP request with the needed data in addition browser dev tools. At some point, I'll explore [`tcpdump`](https://www.tcpdump.org/) and [`wireshark`](https://www.wireshark.org/) to reverse engineer websites for web scraping and share the learnings with you.

If you have anything to share, any questions, suggestions, or something that isn't working correctly, feel free to reach out via Twitter at [@ilyazub_](https://twitter.com/ilyazub_), or [@serp_api](https://twitter.com/serp_api), or Mastodon at [@iz](https://fosstodon.org/@iz@fosstodon.org).


[1]: https://mitmproxy.org/