How to scrape Google search results
Tags: google, seo, tutorial, webscraping
Published: 2024-03-13 13:53:41

---
title: How to scrape Google search results
published: true
date: 2023-11-10 23:00:00 UTC
tags: Google, SEO, Tutorial, Webscraping
canonical_url: https://blog.apify.com/unofficial-google-search-api-from-apify-22a20537a951/
---

 **Hey, we're** [**Apify**](https://apify.it/web-scraping) **, and we've been scraping the web for over 8 years. You can build, deploy, share, and monitor fast, reliable web scrapers on the Apify platform.** [**Check us out**](https://apify.it/web-scraping) **.**

Does Google even need an introduction? If our phones have become extensions of our hands, Google is one of the main reasons for that evolution. Google is a synonym for answers, speed, and accessibility - but also for billions of dollars, users, clicks, searches, and [billions of terabytes](https://lewisdgavin.medium.com/googles-data-footprint-will-blow-your-mind-2237cf8e0d4) of data. That data can be extracted automatically and effortlessly if you know the proper methods and have the right tools at hand.

In this brief how-to article, were going to show you exactly how to scrape the most extensive library in the world by using a ready-made tool on the Apify platform called [Google Search Results Scraper](https://apify.com/apify/google-search-scraper). This is your step-by-step guide to how to scrape any information available from Google, including organic and paid results, ads, queries, People Also Ask boxes, prices, and reviews. Lets get started!

> 🪚 [**Try out Google SERP Scraper**](https://apify.com/apify/google-search-scraper)

## **🕵 What are Google SERPs?**

A Google SERP is a page containing the list of search results that Google displays to you when you type in your query and hit Enter. SERP, in this case, stands for [Search Engine Results Page](https://en.wikipedia.org/wiki/Search_engine_results_page), and youll find SERPs not only on Google, which controls 90% of the search engine market, but also on other search engines, such as Bing, Yahoo, and others. We need to know this term to understand how to use [web scraping](https://blog.apify.com/what-is-web-scraping/) on the Google Search Engine. You can consider the terms `Google page`_,_ `Google search page`_,_ and `Google SERP` to be equal and interchangeable, but well stick with `Google SERP` for the sake of being technically correct.

### **What do Google SERPs look like?**

Google SERPs have changed a lot over the years, with the most prominent features being those infoboxes we all know too well - Knowledge Graphs, Carousels, Featured Snippets - so ubiquitous these days that we cant imagine the Google SERP interface looking any other way. Those now-classic Google SERP features were part of the [Hummingbird algorithm](https://en.wikipedia.org/wiki/Google_Hummingbird) release in 2013.

Its a far cry from the 2003 version of Google results. Does this prehistoric SERP interface ring a bell? Luckily, were not there anymore.

![](https://lh6.googleusercontent.com/C3CGRotvHqIJ-oj7wmHrWZ9KDvYBgCmVxpEQyuV5FQt0tGVMncFrQoZpkvzXjMLcLxD3mDK216dctfeTNFEf7UQ1EKk_JVOsNp7kKAuan41V1klvJa8LV_V5cCCP0ke7-kFC8ZxM=s0)

**_Google SERP interface in the beginning of the 2000s_**

### **How do you scrape Google SERP?**

To scrape Google search results, we first need to understand how Google sees and prioritizes our searches. When you search for things on Google, what you see is not just an index of pages with URLs or so-called `organic searches`. While it used to be like that in the past (as seen above), the primary purpose and driving force of Google - or any search engine for that matter - has always been to have your queries answered as quickly and efficiently as possible, and in a way that will attract your attention but be easy on the eyes.

Thats why over time, Google search results have become much more multilayered, including the results of varying complexity and formats, like a giant layer cake. And that cake-like structure isnt going away any time soon, with voice command search, apps, and mobile search introducing their significant corrections into the way we google stuff. Today, Google search results consist of various levels, depending on the complexity and type of search, as you can see in this example of a _James Webb Space Telescope_ query 🔭

![Search results for James Webb Telescope](https://blog.apify.com/content/images/2022/09/screencapture-google-search-2022-09-07-16_07_35-edit.png)

**_Search results for James Webb Telescope_**

### **What are the elements of a Google search page?**

So these days, Google SERP is packed with various content: featured snippets, so-called snap packs, ads, and organic results. Additional types may also show up: product ads, related searches, and multiple snap pack types (Wikipedia, Google Maps, YouTube videos, etc.)

The elements you receive will depend on the type of search query. For instance, as the James Webb Telescope is rather scientific and educational content, you won't see paid ads there or a carousel with products. But you will see those when googling everyday objects like shoes or headphones. **Leveraging this complicated Google structure means giving access to an incredible amount of useful data, all of which you can scrape.**

## **🔧 How to use data extracted from Google**

Google is the main entry point to the internet for billions of people. This makes appearing in Google Search results a key factor for almost every business. And Google reviews and ratings have a massive impact on local businesses online profiles. Marketing agencies, especially those with a large number of clients from various industries, rely heavily on obtaining reliable [SEO tools](https://www.dealify.com/all-in-one-seo-tools/), including advanced [AI tools](https://simplified.com/ai-tools/). They are not only a means of effectively performing multiple tasks but also a means of successful management and analysis of results. You can look for things like how the top-ranking pages are writing their page titles, the keywords they're targeting, how they format their content, or take it a stage further and do some deeper link analysis.

Typical use cases for Google Search scraping are, among thousands of others:

- **Analyze Google algorithm** and identify its main trends

- **Gain insights for Search engine optimization** (SEO)monitor how your website performs in Google for specific queries over a period of time

- **Analyze ads** ranking for a given set of keywords

- **Monitor competition** in both organic and paid results

- **Build a URL list for specific keywords.** This is useful if you, for example, need good relevant starting points when scraping web pages containing specific phrases.

## **Is it legal to scrape Google?**

Google search results fall into the category of publicly available data, so scraping Google search results is legal. But there is still some data you should not be accumulating, such as personal information or copyrighted content. Learn more about regulations and laws connected to scraping at our [legality article](https://blog.apify.com/is-web-scraping-legal/).

## **🤖 Can I use AI to scrape Google?**

AI is currently unable to scrape websites directly, but it can help generate code for scraping Google if you prompt it with the target elements you want to scrape. Note that the code may not be functional, and website structure and design changes may impact the targeted elements and attributes.

## **🦾 Does Google search have an API?**

Scraping Google search results is how you can create your own Google SERP API (read [What is an API?](https://blog.apify.com/what-is-an-api/) for more info on APIs) to extract data from Google. But why? Technically, you can fish out some insights into the way Google works and displays results without the need to use any specific tools: just google your keyword and see what you get. Now google the same thing in incognito mode and see what you get again. But there are two problems with this approach.

First, the process is pretty **time-consuming** to do manually and at scale - an inefficient monkey job, essentially. Second, the results you get **can't be considered objective** , even if you're using incognito mode. At the beginning of the 2000s, when the Google SERPs were first introduced, they looked much the same to each user for the localized Google version per each country. Now Google algorithms have evolved to really zero in on the most relevant results, so they give out customized results tailored to each user. Google search results these days take into account many factors, such as:

- **Type of device 📱** If a user searches using their smartphone, the search results will look different since, [starting in 2015](https://developers.google.com/search/mobile-sites/mobile-first-indexing), Google prefers showing mobile-optimized web pages.

- **Registration 🔒** If a Google user is logged into their account, what they see on SERPs will be aligned with their history and user behavior, provided that's allowed within their data-related settings.

- **Browser history 📖** If a user rarely empties their browser cache, Google will include that information concerning previous search queries with cookies and adjust the results.

- **Location 📍** If the geolocalization option is activated, Google aligns the SERPs with the user's location. That's why search results for the _sushi takeaway_ query in Prague will be different from those in Los Angeles. If we're talking about local search, the results will be a combination of data from Google Search and Google Maps.

![](https://blog.apify.com/content/images/2022/09/australia.png)

![](https://blog.apify.com/content/images/2022/09/ukraine.png)

**_Just look at how differently Google shows the results for_** hot air balloon **_for users in Australia and Ukraine. Google's algorithm is highly adaptable to location, browser history, and device type._**

### **What about the official Google Search API?**

That's a funny question. Google doesn't provide its own SERP API for web search - so Google doesn't make it that easy to extract data from Google at scale. Moreover, only a limited subset of information available on any search results page can be provided to you via Google services such as [Google Ads](https://ads.google.com/) or [Google Analytics](https://analytics.google.com/analytics/). The two official methods suggested by Google for getting data are Google Custom Search API ([deprecated in April 2018](https://developers.google.com/search/blog/2020/12/search-console-api-updates)) and scraping by [URLFetch method](https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app).

>💡If you're interested in the Google APIs topic in more depth, see our quick guide through Google APIs: [**Top Google search APIs in 2022**](https://blog.apify.com/top-google-search-api/)

## **How do you scrape a search engine?**

So there's a lack of a working solution from Google, a demand to search results objectively, and lots of manual work ahead. The solution to these issues is an automated crawler that is simple enough to use and complex enough to scrape such a massive website as Google. In other words, an alternative SERP API - that's a lot of Caps, but essentially it's a program that will automatically collect data from Google SERP for you to analyze and use. This is precisely what our [Google Search Result Scraper](https://apify.com/apify/google-search-scraper) is created for.

Our SERP API supports the extraction of all data on:

- organic and paid results 🔍

- ads 🛍

- queries 

- People Also Ask 🙋

- prices 🏷

- reviews 

If you need additional attributes, you can also include a short snippet of JavaScript code to extract additional attributes from the HTML. But for now, you can try it out for free on our platform.

> [😉 **If you want a simpler version of Google search results scraper, we have a great alternative for you: Fast Google Search Scraper**](https://apify.com/zuzka/easy-google-scraper)

## **🔍 How to scrape Google search pages**

Now that we've covered all the aspects and reasons for scraping Google let's get started with the tutorial. Promise it won't take long :), but if you would prefer a short video tutorial, we have one for you right here 📼:
{% embed https://youtu.be/eQoO3Wh9JWM %}

**_How to scrape data from Google search pages_**

### **Step 1. Go to Google Search Results Scraper**

Go to the [scraper's page](https://apify.com/apify/google-search-scraper), and click the **Try for free** button. You will be redirected to Apify Console, which is your workspace to run tasks for your scrapers.

![Google Search Results Scraper's page on Apify Store](https://blog.apify.com/content/images/2023/11/free-Google-Search-Results-Scraper-.png)

**_Google Search Results Scraper's page on Apify Store_**

If you are not signed in, you'll find yourself on the sign-up page. [Sign up](https://my.apify.com/sign-in) using your email account, Google, or GitHub - no credit card required. You will be redirected to the scraper's page on your [Apify Console](https://console.apify.com/).

![Log in or sign up with Google, GitHub, or your email address](https://blog.apify.com/content/images/2023/11/google-data-scrape-for-free.png)

**_Log in or sign up with Google, GitHub, or your email address_**

### **Step 2. Insert the keyword you want to scrape**

Now fill in the input fields. You can provide keywords or Google Search URLs as many as you want. Let's see what Google data we'll get for _hot air balloon._ We can use either a search URL or just the keyword; either will work.

![Google Search Results Scraper on Apify Console - typing in the search query](https://blog.apify.com/content/images/2023/11/Google-pages-scraper-free.png)

**_Google Search Results Scraper on Apify Console - typing in the search query_**

### **Step 3. Choose the number of pages for extraction**

Now set how many Google pages you want to scrape and how many results you want to see on each page. So if you are interested in the first 2 pages and 50 results on each, then you'll scrape 100 pages in total. But so is true if you will set the scraper for the first 5 pages and 20 results on each.

![Choose the number of results and Google pages to scrape](https://blog.apify.com/content/images/2023/11/scrape-google-search-pages-1.jpg)

**_Choose the number of results and Google pages to scrape_**

Note that Google will always show you that it has found a gazillion results for almost any query. But in reality, the number of found pages rarely exceeds a few thousand 🤥 This is why it's unlikely that you will find a gazillion pages in your scraped run. You can check it yourself if you set **Results per page** to max on your [Google account](https://www.google.com/preferences?hl=en) settings and see how the _000000_ part of Google pages shrinks to just a few.

### **Step 4. Set up country domain and language of search**

Last but not least - where are we searching? This is your time to specify the domain or country + the language. You can mix&match them as you like (for instance, find all French-speaking results in Canadian Google), but we're going to go with Czech-in-Czech.

![Set up country and language of search. We're going to go for Czech Google in the Czech language 🇨🇿](https://blog.apify.com/content/images/2023/11/Scrape-Google-search-by-domain-and-language-1.png)

**Set up country and language of search. We're going to go for Czech Google in the Czech language 🇨🇿**

### **Step 5. Collect your data extracted from Google search**

Once you are all set, click the **Start** button. Notice that your task will change its status to _Running_, so wait for the scraper's run to finish. It will be just a minute before you see the status switch to _Succeeded_. In our case - 13 seconds, wow! 🚀

Note that we've only got **4 results** which only means the scraper finished extracting data from the Google pages just as we've asked it. **The actual number of results is 100** because we've decided to scrape 2 pages with 50 results each (see **Step 3** ). You can view all results, only organic results or only paid results. You can also preview them as a table or in JSON.

![](https://blog.apify.com/content/images/2023/11/Google-Search-dataset-1.png)

![](https://blog.apify.com/content/images/2023/11/Google-Search-dataset-organic-results.png)

**_After Google Scraper has finished extracting search data, you can preview the results: all, organic and paid._**

You can also preview the data before downloading it by clicking the _Preview in another tab_ 👁 button or viewing it in a new tab if the dataset is too large.

### **Step 6. View and download your Google data**

Once you're ready to download Google data, move to the **Export** button to see the results of your scraping. Now you can download your scraped data in many formats, including HTML table, JSON, CSV, Excel, XML, and RSS feed.

You can choose to download only paid results, only organic results or all of them. Before downloading, you can also narrow down your dataset to a few specific fields that you want to keep or discard. Last option: getting your results [via an API](https://apify.com/apify/google-search-scraper/api/client/nodejs).

![View and download your data in different formats](https://blog.apify.com/content/images/2023/11/View-and-download-your-Google-data-in-CSV--JSON--Excel-and-HTML.png)

**_View and download your data in different formats_**

## **🛡 Do I need proxies for scraping Google SERPs?**

Yes. Apify has proxies designed specifically for SERPs. Our [SERP proxies](https://apify.com/proxy) will make your scraping much faster, and you'll be able to dynamically switch between countries so that you can get search information from any location. If you sign up for a free Apify account, you get a 30-day free trial of our SERP proxy service.

Now that you're all ready go ahead and start your [first month](https://apify.com/pricing) with Apify by using our [Google Search Scraper](https://apify.com/apify/google-search-scraper). If you need to scrape other parts of the Google giant, check out our other Google scrapers at [Apify Store](https://apify.com/store).

### **What are other tools for scraping Google services?**
| 👁 [Google Lens Actor](https://apify.com/alexey/google-lens)            | 📍 [Google Maps Scraper](https://apify.com/compass/crawler-google-places)        |
|--------------------------------|-------------------------------|
| 🔍 [Google Trending Searches](https://apify.com/hooli/google-trending-searches)     | [🛍 Google Shopping Scraper](https://apify.com/emastra/google-shopping-scraper)     |
| 📈 [Google Trends Scraper](https://apify.com/emastra/google-trends-scraper)        | ⭐️ [Google Maps Reviews Scraper](https://apify.com/compass/google-maps-reviews-scraper) |
| 📩📍 [Google Maps Email Extractor](https://apify.com/lukaskrivka/google-maps-with-contact-details)
 | 🤟 [Google Datasets Translator](https://apify.com/hamza.alwan/google-datasets-translator)|