Scrape Yelp Filters, Ad and Organic Results with Python
Tags: webscraping, tutorial, python, programming
Published: 2023-01-28 20:00:47

- <a href="#intro">Intro</a>
- <a href="#what_will_be_scraped">What will be scraped</a>
- <a href="#why_use_api">Why using API?</a>
- <a href="#fullcode">Full Code</a>
- <a href="#preparation">Preparation</a>
- <a href="#code_explanation">Code Explanation</a>
- <a href="#output">Output</a>
- <a href="#links">Links</a>


<h2 id="intro">Intro</h2>

In this blog post, we'll go through the process of extracting [filters](https://serpapi.com/yelp-filters), [organic](https://serpapi.com/yelp-organic-results) and [ad](https://serpapi.com/yelp-ads) results using the [Yelp Search Engine Results API](https://serpapi.com/yelp-search-api) and the Python programming language. You can [look at the complete code in the online IDE (Replit)](https://replit.com/@chukhraiartur/blog-yelp-organic-results#main.py).

<h2 id="what_will_be_scraped">What will be scraped</h2>

![wwbs-yelp-results](https://user-images.githubusercontent.com/81998012/205925658-24a841c4-3b8a-4c47-be20-601afb631bad.png)


<h2 id="why_use_api">Why using API?</h2>

There're a couple of reasons that may use API, ours in particular:

- No need to create a parser from scratch and maintain it.
- Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
- Pay for proxies, and CAPTCHA solvers.
- Don't need to use browser automation.

SerpApi handles everything on the backend with fast response times under ~2.5 seconds (~1.2  seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under [SerpApi Status page](https://serpapi.com/status).

![serpapi-status-all](https://user-images.githubusercontent.com/81998012/206199914-33029b3e-ab80-4ce8-b3f3-5a052bcd8685.png)


<h2 id="fullcode">Full Code</h2>

This code retrieves all the data with pagination:

```python
from serpapi import GoogleSearch
import os, json

params = {
    # https://docs.python.org/3/library/os.html#os.getenv
    'api_key': os.getenv('API_KEY'),    # your serpapi api
    'engine': 'yelp',				    # SerpApi search engine	
    'find_desc': 'Coffee',	            # query
    'find_loc': 'New York, NY, USA',    # location
    'start': 0	     					# pagination
}

search = GoogleSearch(params)           # where data extraction happens on the SerpApi backend
results = search.get_dict()             # JSON -> Python dict

yelp_results = {
    'filters': results['filters'],
    'ads_results': [],
    'organic_results': []
}

while 'error' not in results:
    yelp_results['ads_results'].extend(results['ads_results'])
    yelp_results['organic_results'].extend(results['organic_results'])

    params['start'] += 10
    results = search.get_dict()
    
print(json.dumps(yelp_results, indent=2, ensure_ascii=False))
```


<h2 id="preparation">Preparation</h2>

Install library:

```lang-none
pip install google-search-results
```

[`google-search-results`](https://github.com/serpapi/google-search-results-python) is a SerpApi API package.


<h2 id="code_explanation">Code Explanation</h2>

Import libraries:

```python
from serpapi import GoogleSearch
import os, json
```

| Library       | Purpose                                                            |
|---------------|--------------------------------------------------------------------|
| `GoogleSearch` | to scrape and parse Google results using [SerpApi](https://serpapi.com/) web scraping library. |
| [`os`](https://docs.python.org/3/library/os.html)| to return environment variable (SerpApi API key) value.  |
| [`json`](https://docs.python.org/3/library/json.html)| to convert extracted data to a JSON object. |


The parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the `params` dictionary:

```python
params = {
    # https://docs.python.org/3/library/os.html#os.getenv
    'api_key': os.getenv('API_KEY'),    # your serpapi api
    'engine': 'yelp',				    # SerpApi search engine	
    'find_desc': 'Coffee',	            # query
    'find_loc': 'New York, NY, USA',    # location
    'start': 0	     					# pagination
}
```

|Parameters|Explanation|
|----------|-----------|
|[`api_key`](https://serpapi.com/yelp-search-api#api-parameters-serpapi-parameters-api-key)| Parameter defines the SerpApi private key to use.|
|[`engine`](https://serpapi.com/yelp-search-api#api-parameters-serpapi-parameters-engine)| Set parameter to `yelp` to use the Yelp API engine.|
|[`find_desc`](https://serpapi.com/yelp-search-api#api-parameters-search-query-find-desc)| Parameter defines the query you want to search. You can use anything that you would use in a regular Yelp search.|
|[`find_loc`](https://serpapi.com/yelp-search-api#api-parameters-geographic-location-find-loc)| Parameter defines from where you want the search to originate. You can use any location you would use in a regular Yelp search.|
|[`start`](https://serpapi.com/yelp-search-api#api-parameters-pagination-start)| Parameter defines the result offset. It skips the given number of results. It's used for pagination. (e.g., `0` (default) is the first page of results, `10` is the 2nd page of results, `20` is the 3rd page of results, etc.).|

📌Note: You can also add other [API Parameters](https://serpapi.com/yelp-search-api#api-parameters).

Then, we create a `search` object where the data is retrieved from the SerpApi backend. In the `results` dictionary we get data from JSON:

```python
search = GoogleSearch(params)   # data extraction on the SerpApi backend
results = search.get_dict()     # JSON -> Python dict
```

At the moment, the `results` dictionary only stores data from 1 page. Before extracting data, the `yelp_results` dictionary is created where this data will be added later. Since the filters are repeated on each subsequent page, you can extract them immediately:

```python
yelp_results = {
    'filters': results['filters'],
    'ads_results': [],
    'organic_results': []
}
```

📌Note: When SerpApi encounters `filters`, we add them to our JSON output as the filters object. We are able to extract their `text` and `values`. You can use filters to pass values to [area parameter `l`](https://serpapi.com/yelp-search-api#api-parameters-geographic-location-l), [category parameter `cflt`](https://serpapi.com/yelp-search-api#api-parameters-advanced-yelp-parameters-cflt), and [filters parameter `attrs`](https://serpapi.com/yelp-search-api#api-parameters-advanced-yelp-parameters-attrs).


To get all results, you need to apply pagination. This is achieved by the following check: while there is no `error` in the `results` object of the current page, we extract the data, increase the `start` parameter by `10` to get the results from next page and update the `results` object with the new page data:

```python
while 'error' not in results:
    # data extraction from current page will be here

    params['start'] += 10
    results = search.get_dict()
```

Extending the `yelp_results['ads_results']` and `yelp_results['organic_results']` list with new data from this page:

```python
yelp_results['ads_results'].extend(results['ads_results'])
yelp_results['organic_results'].extend(results['organic_results'])
# ad_title = results['ads_results'][0]['title']
# ad_link = results['ads_results'][0]['link']
# title = results['organic_results'][0]['title']
# rating = results['organic_results'][0]['rating']
# reviews = results['organic_results'][0]['reviews']
```

📌Note: In the comments above, I showed how to extract specific fields. You may have noticed the `results['organic_results'][0]`. This is the index of a organic result, which means that we are extracting data from the first organic result. The `results['organic_results'][1]` is from the second organic result and so on.


After the all data is retrieved, it is output in JSON format:

```python
print(json.dumps(yelp_results, indent=2, ensure_ascii=False))
```


<h2 id="output">Output</h2>

```json
{
  "filters": {
    "neighborhoods": {
      "value": "p:NY:New_York:",
      "list": [
        {
          "text": "Lindenwood",
          "value": "Queens:Lindenwood"
        },
        ... other neighborhoods results
      ]
    },
    "distance": [
      {
        "text": "Bird's-eye View",
        "value": "g:-74.09660339355469,40.62750334315296,-73.89198303222656,40.783660996197945"
      },
      ... other distance results
    ],
    "price": [
      {
        "text": "$",
        "value": "RestaurantsPriceRange2.1"
      },
      ... other price results
    ],
    "category": [
      {
        "text": "Wine Bars",
        "value": "wine_bars"
      },
      ... other category results
    ],
    "features": [
      {
        "text": "Accepts Apple Pay",
        "value": "BusinessAcceptsApplePay"
      },
      ... other features results
    ]
  },
  "ads_results": [
    {
      "block_position": "top",
      "place_ids": [
        "tXWkZsgqEnAGhMJNquO7jQ",
        "dunkin-new-york-131"
      ],
      "title": "Dunkin’",
      "link": "https://www.yelp.com/adredir?ad_business_id=tXWkZsgqEnAGhMJNquO7jQ&amp;campaign_id=rgGYFTjiALhOVokQAQZquQ&amp;click_origin=search_results&amp;placement=above_search&amp;placement_slot=0&amp;redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz%2Fdunkin-new-york-131&amp;request_id=262d581c288008e6&amp;signature=0b160efed560693840ac4428c671613788a9b899a4e0ffc35372dea545ec734f&amp;slot=0",
      "reviews_link": "https://serpapi.com/search.json?engine=yelp_reviews&place_id=tXWkZsgqEnAGhMJNquO7jQ",
      "categories": [
        {
          "title": "Coffee & Tea",
          "link": "https://www.yelp.com/search?cflt=coffee&amp;find_loc=New+York%2C+NY"
        }
      ],
      "reviews": 23,
      "neighborhoods": "Civic Center",
      "offer_details": {
        "title": "Dunkin' Signature Latte",
        "description": "Espresso-Rich Holiday Signature Lattes"
      },
      "phone": "+1-212-732-0406",
      "service_options": {
        "delivery": true,
        "takeout": true
      },
      "button": {
        "text": "Learn More",
        "link": "https://www.yelp.com/adredir?ad_business_id=tXWkZsgqEnAGhMJNquO7jQ&amp;campaign_id=rgGYFTjiALhOVokQAQZquQ&amp;click_origin=search_results&amp;placement=above_search&amp;placement_slot=0&amp;redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz%2Fdunkin-new-york-131&amp;request_id=262d581c288008e6&amp;signature=0b160efed560693840ac4428c671613788a9b899a4e0ffc35372dea545ec734f&amp;slot=0&amp;cta_value=Learn More"
      },
      "thumbnail": "https://s3-media0.fl.yelpcdn.com/offerphoto/EmzkT8VNzsDW0FyXnJu_xQ/ls.jpg"
    },
    ... other ads results
  ],
  "organic_results": [
    {
      "position": 1,
      "place_ids": [
        "ED7A7vDdg8yLNKJTSVHHmg",
        "arabica-brooklyn"
      ],
      "title": "% Arabica",
      "link": "https://www.yelp.com/biz/arabica-brooklyn?osq=Coffee",
      "reviews_link": "https://serpapi.com/search.json?engine=yelp_reviews&place_id=ED7A7vDdg8yLNKJTSVHHmg",
      "categories": [
        {
          "title": "Coffee & Tea",
          "link": "https://www.yelp.com/search?cflt=coffee&amp;find_loc=New+York%2C+NY"
        }
      ],
      "price": "$$",
      "rating": 4.3,
      "reviews": 182,
      "neighborhoods": "Brooklyn Heights",
      "phone": "(718) 865-2551",
      "snippet": "Great coffee had a Spanish latte... can't get over the view! This is the second time we are in Brooklyn and will definitely be back for a 3rd time",
      "service_options": {
        "outdoor_seating": true
      },
      "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/kJkYHT4Q9O5daai-x7paXA/348s.jpg"
    },
    ... other organic results
  ]
}
```

📌Note: Head to the [playground](https://serpapi.com/playground?engine=yelp) for a live and interactive demo.


<h2 id="links">Links</h2>

- [Code in the online IDE](https://replit.com/@chukhraiartur/blog-yelp-organic-results#main.py)
- [Yelp Search Engine Results API](https://serpapi.com/yelp-search-api)
- [Yelp Filters API](https://serpapi.com/yelp-filters)
- [Yelp Ad Results API](https://serpapi.com/yelp-ads)
- [Yelp Organic Results API](https://serpapi.com/yelp-organic-results)

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>

<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>