Scrape Google Daily Search Trends with Python
Tags: webscraping, tutorial, python, programming
Published: 2022-09-04 09:56:41

- <a href="#what_will_be_scraped">What will be scraped</a>
- <a href="#fullcode">Full Code</a>
- <a href="#preparation">Preparation</a>
- <a href="#code_explanation">Code Explanation</a>
    - <a href="#top_level_code_environment">Top-level code environment</a>
    - <a href="#scroll_page">Scroll page</a>
    - <a href="#scrape_daily_search">Scrape daily search</a>
- <a href="#output">Output</a>


<h2 id="what_will_be_scraped">What will be scraped</h2>

![blog-google-trends-daily-what-will-be-scraped](https://user-images.githubusercontent.com/81998012/186502461-b667a3da-fa26-4d95-8228-152eceb3695e.png)

📌Note: For now, we don't have an API that supports extracting data from Google Realtime Search Trends. 

This blog post is to show you way how you can do it yourself while we're working on releasing our proper API in a meantime. We'll update you on our Twitter once this API will be released. 

<h2 id="fullcode">Full Code</h2>

If you don't need explanation, have a look at [full code example in the online IDE](https://replit.com/@chukhraiartur/blog-google-trends-daily-search#main.py).

```python
import time, json
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from parsel import Selector


def scroll_page(url):
	service = Service(ChromeDriverManager().install())
	
	options = webdriver.ChromeOptions()
	options.headless = True
	options.add_argument("--lang=en")
	options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36")
	options.add_argument("--no-sandbox")
	options.add_argument("--disable-dev-shm-usage")
	
	driver = webdriver.Chrome(service=service, options=options)
	driver.get(url)
	
	WebDriverWait(driver, 10000).until(EC.visibility_of_element_located((By.TAG_NAME, 'body')))
	
	flag = True
	
	while flag:
		try:
			search_input = driver.find_element(By.CSS_SELECTOR, 'div[class*="feed-load-more-button"]')
			driver.execute_script("arguments[0].click();", search_input)
			time.sleep(2)
		except:
			flag = False
	
	selector = Selector(driver.page_source)
	driver.quit()
	
	return selector
	
	
def scrape_daily_search(selector):	
	daily_search_trends = {}

	for date in selector.css('.feed-list-wrapper'):
		date_published = date.css('.content-header-title::text').get()
		daily_search_trends[date_published] = []
		
		for item in date.css('.feed-item-header'):
			index = item.css('.index::text').get().strip()
			title = item.css('.title span a::text').get().strip()
			title_link = f"https://trends.google.com{item.css('.title span a::attr(href)').get()}"
			subtitle = item.css('.summary-text a::text').get()
			subtitle_link = item.css('.summary-text a::attr(href)').get()
			source = item.css('.source-and-time span::text').get().strip()
			time_published = item.css('.source-and-time span+ span::text').get().strip()
			searches = item.css('.subtitles-overlap div::text').get().strip()
			image_source = item.css('.image-text::text').get()
			image_source_link = item.css('.image-link-wrapper a::attr(href)').get()
			thumbnail = item.css('.feed-item-image-wrapper img::attr(src)').get()
				
			daily_search_trends[date_published].append({
				'index': index,
				'title': title,
				'title_link': title_link,
				'subtitle': subtitle,
				'subtitle_link': subtitle_link,
				'source': source,
				'time_published': time_published,
				'searches': searches,
				'image_source': image_source,
				'image_source_link': image_source_link,
				'thumbnail': thumbnail,
			})
	
	print(json.dumps(daily_search_trends, indent=2, ensure_ascii=False))


def main():
	GEO = "US"
	URL = f"https://trends.google.com/trends/trendingsearches/daily?geo={GEO}"
	result = scroll_page(URL)
	scrape_daily_search(result)


if __name__ == "__main__":
    main()
```

<h2 id="preparation">Preparation</h2>

**Install libraries**:

```lang-none
pip install parsel selenium webdriver webdriver_manager
```

**Basic knowledge scraping with CSS selectors**

CSS selectors declare which part of the markup a style applies to thus allowing to extract data from matching tags and attributes.

If you haven't scraped with CSS selectors, there's a dedicated blog post of mine
about [how to use CSS selectors when web-scraping](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/) that covers what it is, pros and cons, and why they matter from a web-scraping perspective.

**Reduce the chance of being blocked**

Make sure you're using [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers) [`user-agent`](https://developer.mozilla.org/en-US/docs/Glossary/User_agent) to act as a "real" user visit. Because default `requests` `user-agent` is [`python-requests`](https://github.com/psf/requests/blob/589c4547338b592b1fb77c65663d8aa6fbb7e38b/requests/utils.py#L808-L814) and websites understand that it's most likely a script that sends a request. [Check what's your `user-agent`](https://www.whatismybrowser.com/detect/what-is-my-user-agent/).

There's a [how to reduce the chance of being blocked while web scraping blog post](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/) that can get you familiar with basic and more advanced approaches.


<h2 id="code_explanation">Code Explanation</h2>

Import libraries:

```python
import time, json
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from parsel import Selector
```

| Library       | Purpose                                                            |
|---------------|--------------------------------------------------------------------|
| [`time`](https://docs.python.org/3/library/time.html)    | to work with time in Python.                                  |
| [`json`](https://docs.python.org/3/library/json.html)        | to convert extracted data to a JSON object.                        |
| [`webdriver`](https://www.selenium.dev/documentation/webdriver/)    | to drive a browser natively, as a user would, either locally or on a remote machine using the Selenium server.                                  |
| [`Service`](https://www.selenium.dev/selenium/docs/api/py/webdriver_chrome/selenium.webdriver.chrome.service.html)    | to manage the starting and stopping of the ChromeDriver.                                  |
| [`By`](https://www.selenium.dev/selenium/docs/api/py/webdriver/selenium.webdriver.common.by.html)    | to set of supported locator strategies (By.ID, By.TAG_NAME, By.XPATH etc).                                  |
| [`WebDriverWait`](https://www.selenium.dev/selenium/docs/api/py/webdriver_support/selenium.webdriver.support.wait.html#selenium.webdriver.support.wait.WebDriverWait)    |  to wait only as long as required.                                  |
| [`expected_conditions`](https://www.selenium.dev/selenium/docs/api/py/webdriver_support/selenium.webdriver.support.wait.html#selenium.webdriver.support.wait.WebDriverWait)      | contains a set of predefined conditions to use with WebDriverWait. |
| [`Selector`](https://parsel.readthedocs.io/en/latest/)      | XML/HTML parser that have full [XPath](https://en.wikipedia.org/wiki/XPath) and CSS selectors support. |


<h3 id="top_level_code_environment">Top-level code environment</h3>

This code uses the generally accepted rule of using the [`__name__ == "__main__"`](https://docs.python.org/3/library/__main__.html#) construct: 

```python
def main():
	GEO = "US"
	URL = f"https://trends.google.com/trends/trendingsearches/daily?geo={GEO}"
	result = scroll_page(URL)
	scrape_daily_search(result)


if __name__ == "__main__":
    main()
```

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video [Python Tutorial: if __name__ == '__main__'](https://www.youtube.com/watch?v=sugvnHA7ElY&t=1s) for more details.


A small description of the `main` function:

![daily search](https://user-images.githubusercontent.com/81998012/186505156-809f338f-b0c5-4cb9-830d-9438d9726f14.png)


<h3 id="scroll_page">Scroll page</h3>

The function takes the URL and returns a full HTML structure.

First, let's understand how pagination works on the _daily search trends_ page. To download more information, you must click on the _LOAD MORE_ button below:

![load more](https://user-images.githubusercontent.com/81998012/186506069-8c26d9a4-167e-4be5-876c-7fbb6d2aa5a2.png)

📌Note: To get all the data, you need to press the button until the data runs out.

In this case, [`selenium` library](https://www.selenium.dev/documentation/webdriver/) is used, which allows you to simulate user actions in the browser. For `selenium` to work, you need to use [`ChromeDriver`](https://chromedriver.chromium.org/downloads), which can be downloaded manually or using code. In our case, the second method is used. To control the start and stop of `ChromeDriver`, you need to use [`Service`](https://www.selenium.dev/selenium/docs/api/py/webdriver_chrome/selenium.webdriver.chrome.service.html) which will install browser binaries under the hood:

```python
service = Service(ChromeDriverManager().install())
```

You should also add `options` to work correctly:

```python
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument('--lang=en')
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36")
```

| Chrome options                                                                     | Explanation                                           |
|------------------------------------------------------------------------------|---------------------------------------------------|
| `--headless` | to run Chrome in headless mode. |
| `--lang=en` | to set the browser language to English. |
| `user-agent` | to act as a "real" user request from the browser by passing it to [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers). [Check what's your `user-agent`](https://www.whatismybrowser.com/detect/what-is-my-user-agent). |

Now we can start [`webdriver`](https://www.selenium.dev/documentation/webdriver/) and pass the url to the [`get()` method](https://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webdriver.WebDriver.get).

```python
driver = webdriver.Chrome(service=service, options=options)
driver.get(url)
```

Sometimes it is difficult to calculate how long it will take to load a page, it all depends on the speed of the Internet, the power of the computer and other factors. The method described below is much better than using a delay in seconds since the wait occurs exactly until the moment when the page is fully loaded:

```python
WebDriverWait(driver, 10000).until(EC.visibility_of_element_located((By.TAG_NAME, 'body')))
```

📌Note: In this case, we give 10 seconds for the page to load, if it loads earlier then the wait will end.

When the page has loaded, it is necessary to find the `LOAD MORE` button. Selenium provides the ability to [find element by CSS Selectors](https://selenium-python.readthedocs.io/locating-elements.html#locating-elements-by-css-selectors). 

Clicking the button is done by pasting the JavaScript code into the [`execute_script()`](https://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webdriver.WebDriver.execute_script) method. Wait a while for the data to load using the [`sleep()`](https://docs.python.org/3/library/time.html#time.sleep) method. These actions are repeated as long as the button exists and allows you to download data.

```python
flag = True

while flag:
    try:
        search_input = driver.find_element(By.CSS_SELECTOR, 'div[class*="feed-load-more-button"]')
        driver.execute_script("arguments[0].click();", search_input)
        time.sleep(2)
    except:
        flag = False
```

Now we will use the [`Selector`](https://parsel.readthedocs.io/en/latest/) from the [`Parsel Library`](https://parsel.readthedocs.io/en/latest/) where we pass the `html` structure with all the data, taking into account pagination.

The `parsel` has much faster scraping times because of the engine itself and there is no network component anymore, no real-time interaction with a page and the element, there is only HTML parsing involved. 

After all the operations done, stop the driver:

```python
selector = Selector(driver.page_source)
# extracting code from HTML
driver.quit()
```

The function looks like this:

```python
def scroll_page(url):
	service = Service(executable_path="chromedriver")
	
	options = webdriver.ChromeOptions()
	options.headless = True
	options.add_argument("--lang=en")
	options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36")
	options.add_argument("--no-sandbox")
	options.add_argument("--disable-dev-shm-usage")
	
	driver = webdriver.Chrome(service=service, options=options)
	driver.get(url)
	
	WebDriverWait(driver, 10000).until(EC.visibility_of_element_located((By.TAG_NAME, 'body')))
	
	flag = True
	
	while flag:
		try:
			search_input = driver.find_element(By.CSS_SELECTOR, 'div[class*="feed-load-more-button"]')
			driver.execute_script("arguments[0].click();", search_input)
			time.sleep(2)
		except:
			flag = False
	
	selector = Selector(driver.page_source)
	driver.quit()
	
	return selector
```

In the gif below, I demonstrate how this function works:

![blog-button-clicking-daily](https://user-images.githubusercontent.com/81998012/186656020-9580e2d0-a044-4f34-90fc-35c5cb322ba1.gif)


<h3 id="scrape_daily_search">Scrape daily search</h3>

This function takes a full HTML structure and prints all results in json format.

The data is extracted in such a way that it corresponds to the date publication. 

First of all, you need to find a container that contains data for a specific date publication. We iterate over each date in the loop using `.feed-list-wrapper` the container selector. Each publication date has its own number of items, which also be iterated in a loop using `.feed-item-header` the item selector.

The complete function to scrape all data would look like this:

```python
def scrape_daily_search(selector):	
	daily_search_trends = {}

	for date in selector.css('.feed-list-wrapper'):
		date_published = date.css('.content-header-title::text').get()
		daily_search_trends[date_published] = []
		
		for item in date.css('.feed-item-header'):
			index = item.css('.index::text').get().strip()
			title = item.css('.title span a::text').get().strip()
			title_link = f"https://trends.google.com{item.css('.title span a::attr(href)').get()}"
			subtitle = item.css('.summary-text a::text').get()
			subtitle_link = item.css('.summary-text a::attr(href)').get()
			source = item.css('.source-and-time span::text').get().strip()
			time_published = item.css('.source-and-time span+ span::text').get().strip()
			searches = item.css('.subtitles-overlap div::text').get().strip()
			image_source = item.css('.image-text::text').get()
			image_source_link = item.css('.image-link-wrapper a::attr(href)').get()
			thumbnail = item.css('.feed-item-image-wrapper img::attr(src)').get()
				
			daily_search_trends[date_published].append({
				'index': index,
				'title': title,
				'title_link': title_link,
				'subtitle': subtitle,
				'subtitle_link': subtitle_link,
				'source': source,
				'time_published': time_published,
				'searches': searches,
				'image_source': image_source,
				'image_source_link': image_source_link,
				'thumbnail': thumbnail,
			})
	
	print(json.dumps(daily_search_trends, indent=2, ensure_ascii=False))
```

|Code|Explanation|
|----|-----------|
|`realtime_search_trends`| a temporary `list` where extracted data will be [appended](https://www.w3schools.com/python/ref_list_append.asp) at the end of the function.|
|[`css()`](https://parsel.readthedocs.io/en/latest/usage.html#using-selectors)| to access elements by the passed selector.|
|[`::text` or `::attr(<attribute>)`](https://github.com/scrapy/parsel/blob/90397dcd0b2c1cbb91e44f65c50f9e11628ba028/parsel/csstranslator.py#L48-L51)| to extract textual or attribute data from the node. |
|[`get()`](https://parsel.readthedocs.io/en/latest/usage.html#usage)| to actually extract the textual data.|
|[`strip()`](https://docs.python.org/3/library/stdtypes.html#str.strip)| to return a copy of the string with the leading and trailing characters removed.|
|[`"".join()`](https://docs.python.org/3/library/stdtypes.html#str.join)| to concatenate a list into a string.|
|`realtime_search_trends.append({})`| to [`append` extracted data to a `list`](https://www.w3schools.com/python/ref_list_append.asp) as a dictionary.|

<h2 id="output">Output</h2>

```json
{
  "Wednesday, August 24, 2022": [
    {
      "index": "1",
      "title": "Len Dawson",
      "title_link": "https://trends.google.com/trends/explore?q=Len+Dawson&date=now+7-d&geo=US",
      "subtitle": "Len Dawson, Kansas City Chiefs quarterback and broadcasting ...",
      "subtitle_link": "https://www.npr.org/2022/08/24/1117595982/hall-of-fame-kansas-city-chiefs-quarterback-len-dawson-died-kmbc-kc-mvp",
      "source": "NPR",
      "time_published": "6h ago",
      "searches": "100K+ searches",
      "image_source": "NPR",
      "image_source_link": "https://www.npr.org/2022/08/24/1117595982/hall-of-fame-kansas-city-chiefs-quarterback-len-dawson-died-kmbc-kc-mvp",
      "thumbnail": "https://t1.gstatic.com/images?q=tbn:ANd9GcThi6H-kzkAMrCuhaFZN06AKg24SZAeRr8Wy_tWw_oxJSSO3aqkaR9O3OAJFEmSqPR-lgz-sS__"
    },
    ... other results
    {
      "index": "9",
      "title": "DualSense Edge",
      "title_link": "https://trends.google.com/trends/explore?q=DualSense+Edge&date=now+7-d&geo=US",
      "subtitle": "Everything we know about Sony's modular DualSense Edge ...",
      "subtitle_link": "https://www.inverse.com/gaming/ps5-dualsense-edge-release-date-price-features",
      "source": "Inverse",
      "time_published": "2h ago",
      "searches": "20K+ searches",
      "image_source": "Inverse",
      "image_source_link": "https://www.inverse.com/gaming/ps5-dualsense-edge-release-date-price-features",
      "thumbnail": "https://t0.gstatic.com/images?q=tbn:ANd9GcQ_G2RoFCa44x7fGSpUVCsu5yDzWUq1OdCYIg9qnr5H1oVVWzp6byVRAd4D553fee90tukWKxBl"
    }
  ],
  ... other dates
  "Tuesday, July 26, 2022": [
    {
      "index": "1",
      "title": "Mega Millions",
      "title_link": "https://trends.google.com/trends/explore?q=Mega+Millions&date=today+1-m&geo=US",
      "subtitle": "Mega Millions Friday Jackpot Over a BILLION, What Are The Odds?",
      "subtitle_link": "https://www.focusdailynews.com/mega-millions-crosses-a-billion-what-are-the-odds/",
      "source": "Focusdailynews",
      "time_published": "4w ago",
      "searches": "5M+ searches",
      "image_source": "Focusdailynews",
      "image_source_link": "https://www.focusdailynews.com/mega-millions-crosses-a-billion-what-are-the-odds/",
      "thumbnail": "https://t3.gstatic.com/images?q=tbn:ANd9GcQsSZUkbfXa7ODuRdREyqlZzXU8-_uOnrRybGgPLsvimItLolXz2QCtVIqe6Z1VMITZ_aJJsCAg"
    },
    ... other results
    {
      "index": "16",
      "title": "Alex Jones",
      "title_link": "https://trends.google.com/trends/explore?q=Alex+Jones&date=today+1-m&geo=US",
      "subtitle": "Second day of Alex Jones trial ends in turmoil as lawyers square off",
      "subtitle_link": "https://www.statesman.com/story/news/local/2022/07/27/alex-jones-trial-sandy-hook-defamation-day-2-recap/65385054007/",
      "source": "Austin American-Statesman",
      "time_published": "3w ago",
      "searches": "20K+ searches",
      "image_source": "Austin American-Statesman",
      "image_source_link": "https://www.statesman.com/story/news/local/2022/07/27/alex-jones-trial-sandy-hook-defamation-day-2-recap/65385054007/",
      "thumbnail": "https://t2.gstatic.com/images?q=tbn:ANd9GcT-vdMBiHFcQiLklLHCMZujakKrhEPH7eLmnzlN70b1cn7rJk8MIfpw8UFG2xkEddC7_qs8Wj60"
    }
  ]
}
```

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>

<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>