Scraping the full snippet from Google search result
Tags: webscraping, go
Published: 2024-01-02 00:28:59

Sometimes, you see truncated text on a Google search result like this (...) . Google doesn't always display the meta description of a website. Sometimes, it gets a snippet of relevant text to the search query, which could truncate the text.

![Example of truncated text result](https://serpapi.com/blog/content/images/2023/12/CleanShot-2023-12-29-at-09.24.45@2x.png)

Wonder how you can get the entire snippet of this search result? Let's dive in!

The Idea
--------

The idea is to visit the page URL and scrape part of the relevant text until the next period sign or the whole paragraph.

But before that, we need to find the Google search results list. Therefore, we will use [Google search API by SerpApi](https://serpapi.com/search-api) to scrape the Google SERP.

_You can use any programming language you want, but I'll use Go Lang for this sample._


Scraping Google SERP list with Go lang
--------------------------------------

First, let's get the Google organic results.

### Step 1:

Get your SerpApi key

*   Sign up for free at [SerpApi](https://serpapi.com/).
*   Get your SerpApi Api Key from [this page](https://serpapi.com/manage-api-key).

### Step 2:

Create a new Go Lang project

```
mkdir fullsnippet && cd fullsnippet // Create a new folder and move 
touch main.go // Create a new go file
```


### Step 3:

Install [Golang SerpApi package](https://serpapi.com/blog/scraping-the-full-snippet-from-google-search-result/go%20get%20-u%20github.com/serpapi/serpapi-golang)

```
go mod init project-snippet // Initialize Go Module
go get -u github.com/serpapi/serpapi-golang // Install Go lang package by SerpApi
```


### Step 4:

This is how to get the organic\_results from Google SERP

```
package main

import (
	"fmt"
	"github.com/serpapi/serpapi-golang"
)

const API_KEY = "YOUR_API_KEY"

func main() {
	client_parameter := map[string]string{
		"engine": "google",
		"api_key": API_KEY,
	}
	client := serpapi.NewClient(client_parameter)

	parameter := map[string]string{ 
		"q": "why the sky is blue", // Feel free to change with any keyword
	}

	data, err := client.Search(parameter)
	fmt.Println(data["organic_results"])

	if err != nil {
		fmt.Println(err)
	}
}
```


We've received each result's title, description, link, and other information.

**Collect only specific data**

We can collect and display only specific data in a variable like this

```
data, err := client.Search(parameter)

type OrganicResult struct {
	Title string
	Snippet string
	Link string
}

var organic_results []OrganicResult

for _, result := range data["organic_results"].([]interface{}) {
	result := result.(map[string]interface{})
	organic_result := OrganicResult{
		Title: result["title"].(string),
		Snippet: result["snippet"].(string),
		Link: result["link"].(string),
	}

	organic_results = append(organic_results, organic_result)
}

```


Scraping the individual page
----------------------------

SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use [GoColly package](https://github.com/gocolly/colly).

**Install package**

```
go get -u github.com/gocolly/colly/v2
```


**Code for scraping individual site**  
Add this code inside the loop

```
// Scrape each of the link
c := colly.NewCollector()

c.OnHTML("body", func(e *colly.HTMLElement) {
	rawText := e.Text
	fmt.Println("Raw text in entire body tag:", rawText)
})

// Handle any errors
c.OnError(func(r *colly.Response, err error) {
	fmt.Println("Request URL:", r.Request.URL, "failed with response:", r, "\nError:", err)
})

// Start scraping
c.Visit(organic_result.Link)
```


**If you need the whole text of each site, you can return the `rawText` from above. Then you're done.**

_But if you need only the snippet part until the next period (the entire sentence), we will continue to the following function._

Scraping only the relevant text
-------------------------------

Here's the pseudocode on returning only the relevant full snippet.

```
Find $partialSnippet in the rawText
Find the position of $partialSnippet
Find the next (closest) period after that partial snippet
Return the whole snippet
```


Here is the Go Lang code

```
fullSnippet := findSentence(rawText, snippet)
return fullSnippet
```


The `findSentence` method

```
func findSentence(rawText string, searchText string) string {
	
    // 1. Replace all whitespaces with a single space
	re := regexp.MustCompile(`\s+`) 
	fullText := re.ReplaceAllString(rawText, " ")

	// 2. Replace all backtik ’ into ' at rawText
	re1 := regexp.MustCompile(`’`)
	fullText = re1.ReplaceAllString(fullText, "'")

    // 3. Find the start index of searchText
    startIndex := strings.Index(fullText, searchText)
    if startIndex == -1 {
        return "Text not found"
    }

    // 4. Calculate the end index of the snippet
    snippetEndIndex := startIndex + len(searchText)

    // 5. Find the end of the sentence after the snippet
    endOfSentenceIndex := strings.Index(fullText[snippetEndIndex:], ".")
    if endOfSentenceIndex == -1 {
        // Return the rest of the text from snippet if not found
        return fullText[startIndex:]
    }

    // Adjust to get the correct index in the full text
    endOfSentenceIndex += snippetEndIndex + 1
    
    return fullText[startIndex:endOfSentenceIndex]
}
```


Here is the result

![Result of a full snippet from Google](https://serpapi.com/blog/content/images/2024/01/CleanShot-2024-01-01-at-08.43.47@2x.png)

_You can create a conditional logic (if statement) to only perform this when the snippet has "..." (three dots in the end)._

Full code sample
----------------

Here is the full code sample in GitHub: [https://github.com/hilmanski/serpapi-fullsnippet-golang](https://github.com/hilmanski/serpapi-fullsnippet-golang)

Warning
-------

Here are a few potential issues and solutions with our method.

### Different snippet format

This might not work when Google displays a snippet list, where the snippet comes from some headings or key points. We'll need to write a different logic for this.

### Adding proxy

To prevent getting blocked when scraping the individual site, you can add proxies to the GoColly.

> As a reminder, You don't need to worry about getting block for scraping the Google search itself when using SerpApi.

Reference: [https://go-colly.org/docs/examples/proxy\_switcher/](https://go-colly.org/docs/examples/proxy_switcher/)

Sample proxy switcher

```
package main

import (
	"bytes"
	"log"

	"github.com/gocolly/colly"
	"github.com/gocolly/colly/proxy"
)

func main() {
	// Instantiate default collector
	c := colly.NewCollector(colly.AllowURLRevisit())

	// Rotate two socks5 proxies
	rp, err := proxy.RoundRobinProxySwitcher("socks5://127.0.0.1:1337", "socks5://127.0.0.1:1338")
	if err != nil {
		log.Fatal(err)
	}
	c.SetProxyFunc(rp)

	// Print the response
	c.OnResponse(func(r *colly.Response) {
		log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
	})

	// Fetch httpbin.org/ip five times
	for i := 0; i < 5; i++ {
		c.Visit("https://httpbin.org/ip")
	}
}
```


I hope it helps you to collect more data for your Google SERP!