Web scraping Walmart Search with Nodejs
Tags: webscraping, node, serpapi
Published: 2023-01-26 18:19:33

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/208299658-f2dfa6e5-c4a1-4a94-87b2-b9f38a27ab1a.png)

<h2 id='full_code'>Full code</h2>

If you don't need an explanation, have a look at the [full code example in the online IDE](https://replit.com/@MikhailZub/Scrape-Walmart-Search-with-NodeJS-SerpApi#index.js)

```javascript
import dotenv from "dotenv";
import { config, getJson } from "serpapi";

dotenv.config();
config.api_key = process.env.API_KEY; //your API key from serpapi.com
const resultsLimit = 40; // hardcoded limit for demonstration purpose

const engine = "walmart"; // search engine
const params = {
  query: "macnook pro", // Parameter defines the search query
  page: 1, // Value is used to get the items on a specific page
  device: "desktop", // Parameter defines the device to use to get the results
  store_id: "2280", //Store ID to filter the products by the specific store only
  //other parameters: https://serpapi.com/walmart-search-api#api-parameters
};

const getResults = async () => {
  const results = {
    fixedQuery: null,
    organicResults: [],
  };
  while (results.organicResults.length < resultsLimit) {
    const json = await getJson(engine, params);
    if (!results.fixedQuery) results.fixedQuery = json.search_information?.spelling_fix;
    if (json.organic_results) {
      results.organicResults.push(...json.organic_results);
      params.page += 1;
    } else break;
  }
  return results;
};

getResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='serp_api'>Why use <a href="https://serpapi.com/walmart-search-api">Walmart Search Engine Results API</a> from SerpApi?</h2>

Using API generally solves all or most problems that might get encountered while creating own parser or crawler. From webscraping perspective, our API can help to solve the most painful problems:

- Bypass blocks from supported search engines by solving CAPTCHA or IP blocks.
- No need to create a parser from scratch and maintain it.
- Pay for proxies, and CAPTCHA solvers.
- Don't need to use browser automation if there's a need to extract data in large amounts faster.

Head to the [Playground](https://serpapi.com/playground?engine=walmart) for a live and interactive demo.

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js\* project and add [`npm`](https://www.npmjs.com/) packages [`serpapi`](https://www.npmjs.com/package/serpapi) and [`dotenv`](https://www.npmjs.com/package/dotenv).

To do this, in the directory with our project, open the command line and enter:

```bash
$ npm init -y
```

And then:

```bash
$ npm i serpapi dotenv
```

\*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

- SerpApi package is used to scrape and parse search engine results using SerpApi. Get search results from Google, Bing, Baidu, Yandex, Yahoo, Home Depot, eBay, and more.

- dotenv package is a zero-dependency module that loads environment variables from a `.env` file into `process.env`.

Next, we need to add a top-level "type" field with a value of "module" in our `package.json` file to allow [using ES6 modules in Node.JS](https://nodejs.org/api/packages.html#determining-module-system):

![ES6Module](https://user-images.githubusercontent.com/64033139/206865580-6b5326ee-ff65-4890-876e-78112047a03f.png)

For now, we complete the setup Node.JS environment for our project and move to the step-by-step code explanation.

<h2 id='serp_api_code_explanation'>Code explanation</h2>

First, we need to import `dotenv` from [`dotenv`](https://www.npmjs.com/package/dotenv) library, and `config` and `getJson` from [`serpapi`](https://www.npmjs.com/package/serpapi) library:

```javascript
import dotenv from "dotenv";
import { config, getJson } from "serpapi";
```

Then, we apply some config. Call `dotenv` [`config()`](https://www.npmjs.com/package/dotenv#user-content-config) method, set your SerpApi Private API key to global [`config`](https://github.com/serpapi/serpapi-javascript#configuration) object, and how many results we want to receive (`resultsLimit` constant).

```javascript
dotenv.config();
config.api_key = process.env.API_KEY; //your API key from serpapi.com
const resultsLimit = 40; // hardcoded limit for demonstration purpose
```

- `dotenv.config()` will read your `.env` file, parse the contents, assign it to `process.env`, and return an object with a `parsed` key containing the loaded content or an `error` key if it failed.
- `config.api_key`allows you declare a global `api_key` value by modifying the config object.

Next, we write search `engine` and write the necessary search parameters for making a request ([get the full JSON list of supported Walmart Stores](https://serpapi.com/walmart-stores)):

📌Note: I specifically made a mistake in the search query to demonstrate how [Walmart Spell Check API](https://serpapi.com/walmart-spell-check) works.

```javascript
const engine = "walmart"; // search engine
const params = {
  query: "macnook pro", // Parameter defines the search query
  page: 1, // Value is used to get the items on a specific page
  device: "desktop", // Parameter defines the device to use to get the results
  store_id: "2280", //Store ID to filter the products by the specific store only
};
```

📌Note: Also see [SerpApi Python demo project of extracting data from 500 Walmart stores and analyzing extracted data](https://serpapi.com/blog/serpapi-demo-project-walmart-coffee-exploratory-data-analysis/) if you want to know more about scraping Walmart.

You can see all available parameters in the [API documentation](https://serpapi.com/walmart-search-api#api-parameters).

Next, we declare the function `getResult` that gets data from the page and return it:

```javascript
const getResults = async () => {
  ...
};
```

In this function we need to declare an object two keys: `fixedQuery` is equal to `null`, and empty `organicResults` array, then and using [`while`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/while) loop get `json` with results, add `spelling_fix` to the `fixedQuery` on the first iteration, and add `organic_results` to `organicResults` array ([`push()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/push) method) from each page and set next page index (to `params.page` value).

If there are no more results on the page or if the number of received results is more than `reviewsLimit` we stop the loop (using [`break`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/break)) and return an array with results:

```javascript
const results = {
  fixedQuery: null,
  organicResults: [],
};
while (results.organicResults.length < resultsLimit) {
  const json = await getJson(engine, params);
  if (!results.fixedQuery) results.fixedQuery = json.search_information?.spelling_fix;
  if (json.organic_results) {
    results.organicResults.push(...json.organic_results);
    params.page += 1;
  } else break;
}
return results;
```

And finally, we run the `getResults` function and print all the received information in the console with the [`console.dir`](https://nodejs.org/api/console.html#consoledirobj-options) method, which allows you to use an object with the necessary parameters to change default output options:

```javascript
getResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='serp_api_output'>Output</h2>

```json
{
   "fixedQuery":"macbook pro",
   "organicResults":[
      {
         "us_item_id":"121393924",
         "product_id":"18F5MJ3R95JG",
         "title":"Apple MacBook Air, 13.3-inch, Intel Core i5, 4GB RAM, Mac OS, 128GB SSD, Bundle: Black Case, Wireless Mouse, Bluetooth Headset - Silver",
         "thumbnail":"https://i5.walmartimages.com/asr/60e5ea72-ac15-4bdc-b112-572f76776e83.77df8900f9478a7a581dad9a6698ecd5.jpeg?odnHeight=180&odnWidth=180&odnBg=FFFFFF",
         "rating":3.5,
         "reviews":100,
         "seller_id":"F86EF73A620D4265AEE28E9FD77A4ED1",
         "seller_name":"Certified 2 Day Express",
         "fulfillment_badges":[
            "2-day shipping"
         ],
         "two_day_shipping":false,
         "out_of_stock":false,
         "sponsored":true,
         "muliple_options_available":false,
         "primary_offer":{
            "offer_id":"53A0316C100D4D1EBD2AD8753FC4FE25",
            "offer_price":349,
            "min_price":0
         },
         "price_per_unit":{
            "unit":"each",
            "amount":""
         },
         "product_page_url":"https://www.walmart.com/ip/Apple-MacBook-Air-13-3-inch-Intel-Core-i5-4GB-RAM-Mac-OS-128GB-SSD-Bundle-Black-Case-Wireless-Mouse-Bluetooth-Headset-Silver/121393924",
         "serpapi_product_page_url":"https://serpapi.com/search.json?device=desktop&engine=walmart_product&product_id=121393924"
      },
      ... and other results
   ]
}
```

<h2 id='links'>Links</h2>

- [Code in the online IDE](https://replit.com/@MikhailZub/Scrape-Walmart-Search-with-NodeJS-SerpApi#index.js)
- [Walmart Search Engine Results API Documentation](https://serpapi.com/walmart-search-api)
- [Walmart Search Engine Results API Playground](https://serpapi.com/playground?engine=walmart)

If you want other functionality added to this blog post or if you want to see some projects made with SerpApi, [write me a message](mailto:miha01012019@gmail.com).

---

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>