Scrape Walmart Search for a specific store
Tags: webscraping, walmart
Published: 2021-12-02 11:50:33

---
canonical_url: https://serpapi.com/blog/scrape-walmart-search-results-for-a-specific-store-id-with-plain-http-requests/
description: Update location cookies to specify the location for plain HTTP requests to Walmart.
---

Walmart responds with results for Sacramento for requests outside of the US.

![image](https://user-images.githubusercontent.com/282605/142628959-2feeeb6c-d085-40bb-a282-26c758ce2f31.png)

But how to search for products that are available in a specific store? Any store on Walmart can be chosen without browser automation &mdash; only by setting relevant cookies in the plain HTTP request.

> To figure out on your own, JS and browser dev tools knowledge will be enough. Some Ruby knowledge is required to understand this post. 

## Location cookies

I've updated location several times and checked the browser Dev Tools -> Application -> Cookies.

![image](https://user-images.githubusercontent.com/282605/142677221-555a6ff9-3ff3-4f3f-b02b-1a33645b61d9.png)

There are several cookies being updated after choosing a different location: `locGuestData`, `locDataV3`, `assortmentStoreId`; `ACID`, `hasACID`, `hasLocData`.

`location-data` also looks relevant but it contains postal code and address for a store I haven't chosen. Maybe it was used before Walmart migrated to GrapgQL API.

`locDataV3` and `locGuestData` are Base64 and URI-encoded JSON objects. `locDataV3` contains more data than `locGuestData`. But data of `locGuestData` can be used for both.

`ACID` is a UUID. It can be generated on the client.

`hasACID` and `hasLocData` are flags.

### Understanding `locGuestData`

Let's check what's inside this cookie value to understand how to set the store ID.

#### Example of encoded `locGuestData`

When sending requests to Walmart, `locGuestData` is a Base64-encoded string.

```js
eyJpbnRlbnQiOiJTSElQUElORyIsInN0b3JlSW50ZW50IjoiUElDS1VQIiwibWVyZ2VGbGFnIjp0cnVlLCJwaWNrdXAiOnsibm9kZUlkIjoiNDExNSIsInRpbWVzdGFtcCI6MTYzNzMyODUwMDUyM30sInBvc3RhbENvZGUiOnsiYmFzZSI6Ijc4MTU0IiwidGltZXN0YW1wIjoxNjM3MzI4NTAwNTIzfSwidmFsaWRhdGVLZXkiOiJwcm9kOnYyOjUyNzNlMDFjLTA4NzAtNGUwOS05ODU4LTAzYTI2ZDQ5N2ZhOSJ9
```

#### Example of decoded `locGuestData`

This Base64 string is a encoded JSON object.

```js
JSON.parse(decodeURIComponent(atob("eyJpbnRlbnQiOiJTSElQUElORyIsInN0b3JlSW50ZW50IjoiUElDS1VQIiwibWVyZ2VGbGFnIjp0cnVlLCJwaWNrdXAiOnsibm9kZUlkIjoiNDExNSIsInRpbWVzdGFtcCI6MTYzNzMyODUwMDUyM30sInBvc3RhbENvZGUiOnsiYmFzZSI6Ijc4MTU0IiwidGltZXN0YW1wIjoxNjM3MzI4NTAwNTIzfSwidmFsaWRhdGVLZXkiOiJwcm9kOnYyOjUyNzNlMDFjLTA4NzAtNGUwOS05ODU4LTAzYTI2ZDQ5N2ZhOSJ9")))

{
    "intent": "SHIPPING",
    "storeIntent": "PICKUP",
    "mergeFlag": true,
    "pickup": {
        "nodeId": "4115",
        "timestamp": 1637328500523
    },
    "postalCode": {
        "base": "78154",
        "timestamp": 1637328500523
    },
    "validateKey": "prod:v2:5273e01c-0870-4e09-9858-03a26d497fa9"
}
```

After changing Walmart store several times, I've seen that `nodeId` and `postalCode.base` are changing. 

#### Generate `timestamp` and `acid` for `locGuestData`

`timestamp` and `acid` can be generated on every request.

```ruby
timestamp = Time.now.to_i

acid = SecureRandom.uuid
```

#### Base64-encode location data

Next, let's Base64-encode that JSON string as Walmart expects.

```ruby
timestamp = Time.now.to_i

acid = SecureRandom.uuid

location_guest_data = {
  intent: "SHIPPING",
  storeIntent: "PICKUP",
  mergeFlag: true,
  pickup: {
    nodeId: store_id,
    timestamp: timestamp
  },
  postalCode: {
    base: postal_code,
    timestamp: timestamp
  },
  validateKey: "prod:v2:#{acid}"
}

encoded_location_data = Base64.urlsafe_encode64(JSON.dump(location_guest_data))
```

#### Create cookie string

Finally, a location cookie string contains all the required fields.

```ruby
%(ACID=#{acid}; hasACID=true; hasLocData=1; locDataV3=#{location_guest_data}; assortmentStoreId=#{store_id}; locGuestData=#{encoded_location_data})
```

### Complete function to create Walmart location cookie

Putting all together.

```ruby
def location_cookie(store_id, postal_code)
  return if store_id.blank?

  timestamp = Time.now.to_i

  acid = SecureRandom.uuid

  location_guest_data = {
    intent: "SHIPPING",
    storeIntent: "PICKUP",
    mergeFlag: true,
    pickup: {
      nodeId: store_id,
      timestamp: timestamp
    },
    postalCode: {
      base: postal_code,
      timestamp: timestamp
    },
    validateKey: "prod:v2:#{acid}"
  }

  encoded_location_data = Base64.urlsafe_encode64(JSON.dump(location_guest_data))

  %(ACID=#{acid}; hasACID=true; hasLocData=1; locDataV3=#{location_guest_data}; assortmentStoreId=#{store_id}; locGuestData=#{encoded_location_data})
end
```

Then make an HTTP request using the language and libraries you've chosen.

```js
import got from 'got';

const STORE_ID = "4115";
const POSTAL_CODE = "78154";

const locationCookie = getLocationCookie(STORE_ID, POSTAL_CODE);

const htmlResponse = await got('https://www.walmart.com/search?q=cookie', {
  headers: {
    cookie: locationCookie
  }
});
```

![image](https://user-images.githubusercontent.com/282605/142636729-44dce11c-3566-4bb9-a34a-a4a5988990b2.png)


## Where to get store ID and postal code

Well, but we wouldn't hard-code store ID and postal code into the web scraping program. A [CSV of 4.6k stores](https://raw.githubusercontent.com/ilyazub/walmart-store-locator/master/output/walmart_stores.csv) can be used to find and store ID dynamically.

Programmatic usage of CSV is out of the scope of this post. All that is needed is to read find store ID and postal code for a specific location in a table.

### Updating a list of Walmart stores IDs and locations

Walmart provides several sources to find stores. Data can be populated from one of those sources:
- [Store Finder](https://www.walmart.com/store/finder)
- [US Store Directory](https://www.walmart.com/store/directory)

#### Store Directory

Store Directory contains links on four levels: country, states, cities, and stores. To get the data, iterate over all elements on the specific level and make subsequent requests.

##### States

Assuming the country is the US, 51 states can be hard-coded. Walmart front-end requests data from the JSON endpoint `https://www.walmart.com/store/electrode/api/store-directory`. It accepts the `st` search parameter.

Example: `https://www.walmart.com/store/electrode/api/store-directory?st=AL`.

It returns a list of cities. Each city object contains `city`, and `storeId` or `storeCount`. The city with `storeId` contains a single store. The city with `storeCount` contains multiple stores.

##### Single store in a city

Request to a specific store returns an HTML page. Example: `https://www.walmart.com/store/5744`.

![image](https://user-images.githubusercontent.com/282605/142639733-32c22320-5cfd-4cb9-b744-8a463dd221f5.png)

Store address and postal code should be extracted from the HTML. Store ID is already in URI.

```js
let postalCode = document.querySelector(".store-address-postal[itemprop=postalCode]").textContent;
let address = document.querySelector(".store-address[itemprop=address]").textContent;
```

##### Multiple stores in a city

Request for multiple stores returns a JSON response. Cities with a single store respond with an empty array (`[]`) so we have to parse HTML.

Example request for multiple stores

```
https://www.walmart.com/store/electrode/api/store-directory?st=AL&city=Decatur
```

Sample city from the response

```json
{
  "displayName": "Neighborhood Market",
  "storeName": "Neighborhood Market",
  "address": "1203 6th Ave Se",
  "phone": "256-822-6366",
  "postalCode": "35601",
  "storeId": 2488
}
```

##### Putting all together

Pseudo-code to collect store IDs and locations for all US states.

```js
const STATES = ["AL", "TX", "CA", /* ... */];

let walmartStores = [];

for (let state of STATES) {
  let cities = get(`https://www.walmart.com/store/electrode/api/store-directory?st=${state}`);

  for (let { storeId, storeCount, city } of cities) {
    if (storeId && !storeCount) {
      let store = get(`https://www.walmart.com/store/${storeId}`);

      let document = parseHTML(store);

      let postalCode = document.querySelector(".store-address-postal[itemprop=postalCode]").textContent;
      let address = document.querySelector(".store-address[itemprop=address]").textContent;

      walmartStores.push({ postalCode, address, storeId: storeId });
    } else if (!storeId && storeCount > 0) {
      let stores = get(`https://www.walmart.com/store/electrode/api/store-directory?st=${state}&city=${city}`);

      walmartStores.concat(stores);
    }
  }
}

csv.write("walmart_stores.csv", walmartStores);
```

#### Existing programs to scrape Walmart Stores

[Search on GitHub](https://grep.app/search?q=walmart.com/store/) via grep.app shows four relevant repositories

- [Akamai edge workers example](https://grep.app/search?q=walmart.com/store/). But it contains only 471 Walmart stores.
 
```bash
$ curl -s https://raw.githubusercontent.com/akamai/edgeworkers-examples/master/edgecompute/examples/personalization/storelocator/data/locations.json | jq '.elements[].tags | select(."ref:walmart" != null) | .ref' | wc -l
471
```

- [`scrapehero/walmart_store_locator`](https://github.com/scrapehero/walmart_store_locator/blob/8935e4d743e5507453a2a99294028fa6016cb3c9/walmart_store_locator.py) which scrapes stores by postal codes. But finding a list of actual postal codes turned out to be harder than finding a list of actual US states.

- [`theriley106/WaltonAnalytics`](https://github.com/theriley106/WaltonAnalytics/blob/3a8fabc25ec757d94f5354ff2fc7c239a6c51fc7/Analytics.py) which is great to extract data from Walmart but not Walmart stores.

- [`GUI/covid-vaccine-spotter`](https://github.com/GUI/covid-vaccine-spotter/blob/3995274876a72990ae7699afa6e692b538efe48b/src/providers/Walmart/Stores.js#L129) which scrapes stores by postal codes. But finding a list of actual postal codes turned out to be harder than finding a list of actual US states.

So, I've played with Rust and came up with [this (rough) program](https://github.com/ilyazub/walmart-store-locator).

After going through compilation errors, it worked well. Thanks to this helpful [blog post about async streams in Rust](https://gendignoux.com/blog/2021/04/01/rust-async-streams-futures-part1.html). Every time my program compiled, it actually worked. Fixing compilation errors is hard (for non-rustacean) but there's was no need to debug the program in runtime which is great.

<iframe id="ytplayer" type="text/html" width="640" height="360"
  src="https://www.youtube.com/embed/-d5aU1WC0Ng?autoplay=0&fs=0&iv_load_policy=3&showinfo=0&rel=0&cc_load_policy=0&start=0&end=0&origin=https://serpapi.com/blog"
  frameborder="0" scrolling="no" marginheight="0"></iframe>

## Conclusion

Scraping Walmart is fairly easy &mdash; it contains inline JSON data for all products on the search results page.

![image](https://user-images.githubusercontent.com/282605/142681839-a99751e2-5a81-4cd7-8784-bbbaaec301ec.png)

Update location cookies to specify the location for plain HTTP requests to Walmart.

If you have anything to share, any questions, suggestions, or something that isn't working correctly, feel free to drop a comment in the comment section or reach out via Twitter at [@ilyazub_](https://twitter.com/ilyazub_), or [@serp_api](https://twitter.com/serp_api).

Yours,
Ilya, and the rest of the SerpApi Team.

----

Join us on [Reddit](https://www.reddit.com/r/SerpApi/) | [Twitter](https://twitter.com/serp_api) | [YouTube](https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg)