Web scraping Apple App Store Search with Nodejs
Tags: webscraping, node, serpapi
Published: 2023-01-03 18:14:55

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/206864585-2c7ee8b5-7686-403b-a50f-f2ffa88426e8.png)

📌Note: in this blog post I'll show you how to scrape Apple App Store Search and receive the result exactly like on Apple iMac, because search results on Mac are absolutely different than results on PC. The screenshots below show you the difference:

- Mac results:
  ![Mac](https://user-images.githubusercontent.com/64033139/206864240-6bc521bf-f568-43ce-aaba-41ffb428a81a.png)

- PC results:
  ![PC](https://user-images.githubusercontent.com/64033139/206864493-7e5c037a-6d46-438e-9589-36149223772b.png)

<h2 id='full_code'>Full code</h2>

If you don't need an explanation, have a look at [the full code example in the online IDE](https://replit.com/@MikhailZub/Scrape-Apple-App-Store-Search-with-NodeJS-SerpApi#index.js)

```javascript
import dotenv from "dotenv";
dotenv.config();
import { getJson } from "serpapi";

const engine = "apple_app_store"; // search engine
const resultsLimit = 50; // hardcoded limit for demonstration purpose
const params = {
  api_key: process.env.API_KEY, //your API key from serpapi.com
  term: "image viewer", // Parameter defines the query you want to search
  country: "us", // Parameter defines the country to use for the search
  lang: "en-us", // Parameter defines the language to use for the search
  device: "desktop", //Parameter defines the device to use to get the results. It can be set to "desktop", "tablet", or "mobile" (default)
  num: "10", // Parameter defines the number of results you want to get per each page
  page: 0, // Parameter is used to get the items on a specific page
};

const getResults = async () => {
  const results = [];
  while (true) {
    const json = await getJson(engine, params);
    if (json.organic_results) {
      results.push(...json.organic_results);
      params.page += 1;
    } else break;
    if (results.length >= resultsLimit) break;
  }
  return results;
};

getResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='serp_api'>Why use <a href="https://serpapi.com/yelp-organic-results">Apple App Store Search Scraper API</a> from SerpApi?</h2>

Using API generally solves all or most problems that might get encountered while creating own parser or crawler. From webscraping perspective, our API can help to solve the most painful problems:

- Bypass blocks from supported search engines by solving CAPTCHA or IP blocks.
- No need to create a parser from scratch and maintain it.
- Pay for proxies, and CAPTCHA solvers.
- Don't need to use browser automation if there's a need to extract data in large amounts faster.

Head to the [Playground](https://serpapi.com/playground?engine=apple_app_store) for a live and interactive demo.

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js\* project and add [`npm`](https://www.npmjs.com/) packages [`serpapi`](https://www.npmjs.com/package/serpapi) and [`dotenv`](https://www.npmjs.com/package/dotenv).

To do this, in the directory with our project, open the command line and enter:

```bash
$ npm init -y
```

And then:

```bash
$ npm i serpapi dotenv
```

\*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

- SerpApi package is used to scrape and parse search engine results using SerpApi. Get search results from Google, Bing, Baidu, Yandex, Yahoo, Home Depot, eBay, and more.

- dotenv package is a zero-dependency module that loads environment variables from a `.env` file into `process.env`.

Next, we need to add a top-level "type" field with a value of "module" in our `package.json` file to allow [using ES6 modules in Node.JS](https://nodejs.org/api/packages.html#determining-module-system):

![ES6Module](https://user-images.githubusercontent.com/64033139/206865580-6b5326ee-ff65-4890-876e-78112047a03f.png)

For now, we complete the setup Node.JS environment for our project and move to the step-by-step code explanation.

<h2 id='serp_api_code_explanation'>Code explanation</h2>

First, we need to import `dotenv` from [`dotenv`](https://www.npmjs.com/package/dotenv) library and call [`config()`](https://www.npmjs.com/package/dotenv#user-content-config) method, then import `getJson` from [`serpapi`](https://www.npmjs.com/package/serpapi) library:

```javascript
import dotenv from "dotenv";
dotenv.config();
import { getJson } from "serpapi";
```

- `config()` will read your `.env` file, parse the contents, assign it to `process.env`, and return an Object with a `parsed` key containing the loaded content or an `error` key if it failed.
- `getJson()` allows you to get a JSON response based on search parameters.

Next, we write search `engine`, set how many results we want to receive (`resultsLimit` constant) and write the necessary search parameters for making a request:

```javascript
const engine = "apple_app_store"; // search engine
const resultsLimit = 50; // hardcoded limit for demonstration purpose
const params = {
  api_key: process.env.API_KEY, //your API key from serpapi.com
  term: "image viewer", // Parameter defines the query you want to search
  country: "us", // Parameter defines the country to use for the search
  lang: "en-us", // Parameter defines the language to use for the search
  device: "desktop", //Parameter defines the device to use to get the results. It can be set to "desktop", "tablet", or "mobile" (default)
  num: "10", // Parameter defines the number of results you want to get per each page
  page: 0, // Parameter is used to get the items on a specific page
};
```

You can use the next search params:

- `api_key` parameter defines the SerpApi private key to use.
- `term` parameter defines the query you want to search. You can use any search term that you would use in a regular App Store search.
- `country` parameter defines the country to use for the search. It's a two-letter country code. (e.g., `us` (default) for the United States, `uk` for United Kingdom, or `fr` for France). Head to the [Apple Regions](https://serpapi.com/apple-regions) for a full list of supported Apple Regions.
- `lang` parameter defines the language to use for the search. It's a four-letter country code. (e.g., `en-us` (default) for the English, `fr-fr` for French, or `uk-ua` for Ukranian). Head to the Apple Languages for a full list of supported [Apple Languages](https://serpapi.com/apple-languages).
- `num` parameter defines the number of results you want to get per each page. It defaults to `10`. Maximum number of results you can get per page is `200`. Any number greater than maximum number will default to `200`.
- `page` parameter is used to get the items on a specific page. (e.g., `0` (default) is the first page of results, `1` is the 2nd page of results, `2` is the 3rd page of results, etc.).
- `disallow_explicit` parameter defines the filter for disallowing explicit apps. It defaults to `false`.
- `property` parameter allows to search the property of an app. `developer` allows searching the developer title of an app ( e.g., `property: "developer"` and `term: "Coffee"` gives apps with "Coffee" in their developer's name. (Ex: `Coffee Inc.`).
- `no_cache` parameter will force SerpApi to fetch the App Store Search results even if a cached version is already present. A cache is served only if the query and all parameters are exactly the same. Cache expires after 1h. Cached searches are free, and are not counted towards your searches per month. It can be set to `false` (default) to allow results from the cache, or `true` to disallow results from the cache. `no_cache` and `async` parameters should not be used together.
- `async` parameter defines the way you want to submit your search to SerpApi. It can be set to `false` (default) to open an HTTP connection and keep it open until you got your search results, or `true` to just submit your search to SerpApi and retrieve them later. In this case, you'll need to use our [Searches Archive API](https://serpapi.com/search-archive-api) to retrieve your results. `async` and `no_cache` parameters should not be used together. `async` should not be used on accounts with [Ludicrous Speed](https://serpapi.com/plan) enabled.
- `device` parameter defines the device to use to get the results. It can be set to `desktop` to use a Mac App Store, `tablet` to use an iPad App Store, or `mobile` (default) to use an iPhone App Store.

Next, we declare the function `getResult` that gets data from the page and return it:

```javascript
const getResults = async () => {
  ...
};
```

In this function we need to declare an empty `results` array and using [`while`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/while) loop get `json` with results, add `organic_results` from each page and set next page index (to `params.page` value).

If there is no more results on the page or if the number of received results more than`resultsLimit` we stop the loop (using [`break`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/break)) and return an array with results:

```javascript
const results = [];
while (true) {
  const json = await getJson(engine, params);
  if (json.organic_results) {
    results.push(...json.organic_results);
    params.page += 1;
  } else break;
  if (results.length >= resultsLimit) break;
}
return results;
```

And finally, we run the `getResults` function and print all the received information in the console with the [`console.dir`](https://nodejs.org/api/console.html#consoledirobj-options) method, which allows you to use an object with the necessary parameters to change default output options:

```javascript
getResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='serp_api_output'>Output</h2>

```json
[
  {
    "position": 1,
    "id": 1507782672,
    "title": "Pixea",
    "bundle_id": "imagetasks.Pixea",
    "version": "1.4",
    "vpp_license": true,
    "age_rating": "4+",
    "release_note": "- New icon - macOS Big Sur support - Universal Binary - Bug fixes and improvements",
    "seller_link": "https://www.imagetasks.com",
    "minimum_os_version": "10.12",
    "description": "Pixea is an image viewer for macOS with a nice minimal modern user interface. Pixea works great with JPEG, HEIC, PSD, RAW, WEBP, PNG, GIF, and many other formats. Provides basic image processing, including flip and rotate, shows a color histogram, EXIF, and other information. Supports keyboard shortcuts and trackpad gestures. Shows images inside archives, without extracting them. Supported formats: JPEG, HEIC, GIF, PNG, TIFF, Photoshop (PSD), BMP, Fax images, macOS and Windows icons, Radiance images, Google's WebP. RAW formats: Leica DNG and RAW, Sony ARW, Olympus ORF, Minolta MRW, Nikon NEF, Fuji RAF, Canon CR2 and CRW, Hasselblad 3FR. Sketch files (preview only). ZIP-archives. Export formats: JPEG, JPEG-2000, PNG, TIFF, BMP. Found a bug? Have a suggestion? Please, send it to support@imagetasks.com Follow us on Twitter @imagetasks!",
    "link": "https://apps.apple.com/us/app/pixea/id1507782672?mt=12&uo=4",
    "serpapi_product_link": "https://serpapi.com/search.json?country=us&engine=apple_product&product_id=1507782672&type=app",
    "serpapi_reviews_link": "https://serpapi.com/search.json?country=us&engine=apple_reviews&page=1&product_id=1507782672",
    "release_date": "2020-04-20 07:00:00 UTC",
    "price": {
      "type": "Free"
    },
    "rating": [
      {
        "type": "All Times",
        "rating": 0,
        "count": 0
      }
    ],
    "genres": [
      {
        "name": "Photo & Video",
        "id": 6008,
        "primary": true
      },
      {
        "name": "Graphics & Design",
        "id": 6027,
        "primary": false
      }
    ],
    "developer": {
      "name": "ImageTasks Inc",
      "id": 450316587,
      "link": "https://apps.apple.com/us/developer/id450316587"
    },
    "size_in_bytes": 5838181,
    "supported_languages": ["EN"],
    "screenshots": {
      "general": [
        {
          "link": "https://is3-ssl.mzstatic.com/image/thumb/PurpleSource124/v4/b1/8c/fb/b18cfb80-cb5c-d67d-2edc-ee1f6666e012/35b8d5a7-b493-4a80-bdbd-3e9d564601dd_Pixea-1.jpg/800x500bb.jpg",
          "size": "800x500"
        },
        {
          "link": "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource124/v4/96/08/83/9608834d-3d2b-5c0b-570c-f022407ff5cc/1836573e-1b6a-421c-b654-6ae2f915d755_Pixea-2.jpg/800x500bb.jpg",
          "size": "800x500"
        },
        {
          "link": "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource124/v4/58/fd/db/58fddb5d-9480-2536-8679-92d6b067d285/98e22b63-1575-4ee6-b08d-343b9e0474ea_Pixea-3.jpg/800x500bb.jpg",
          "size": "800x500"
        },
        {
          "link": "https://is2-ssl.mzstatic.com/image/thumb/PurpleSource124/v4/c3/f3/f3/c3f3f3b5-deb0-4b58-4afc-79073373b7b9/28f51f38-bc59-4a61-a5a1-bff553838267_Pixea-4.jpg/800x500bb.jpg",
          "size": "800x500"
        }
      ]
    },
    "logos": [
      {
        "size": "60x60",
        "link": "https://is1-ssl.mzstatic.com/image/thumb/Purple114/v4/73/5f/29/735f2997-66b7-9795-ad4f-7ed78d0d3812/AppIcon-0-0-85-220-0-0-0-0-4-0-0-0-2x-sRGB-0-0-0-0-0.png/60x60bb.png"
      },
      {
        "size": "512x512",
        "link": "https://is1-ssl.mzstatic.com/image/thumb/Purple114/v4/73/5f/29/735f2997-66b7-9795-ad4f-7ed78d0d3812/AppIcon-0-0-85-220-0-0-0-0-4-0-0-0-2x-sRGB-0-0-0-0-0.png/512x512bb.png"
      },
      {
        "size": "100x100",
        "link": "https://is1-ssl.mzstatic.com/image/thumb/Purple114/v4/73/5f/29/735f2997-66b7-9795-ad4f-7ed78d0d3812/AppIcon-0-0-85-220-0-0-0-0-4-0-0-0-2x-sRGB-0-0-0-0-0.png/100x100bb.png"
      }
    ]
  },
  ...and other results
]
```

<h2 id='links'>Links</h2>

- [Code in the online IDE](https://replit.com/@MikhailZub/Scrape-Apple-App-Store-Search-with-NodeJS-SerpApi#index.js)
- [Apple App Store Search Scraper API documentation](https://serpapi.com/apple-app-store)
- [Apple App Store Search Scraper API Playground](https://serpapi.com/playground?engine=apple_app_store)

If you want other functionality added to this blog post or if you want to see some projects made with SerpApi, [write me a message](mailto:miha01012019@gmail.com).

---

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>