SerpApi Async Requests with Pagination using Python
Tags: python, api, tutorial, webscraping
Published: 2022-11-17 11:49:09

- <a href="#intro">Intro</a>
- <a href="#time-comparison">Time Comparison</a>
- <a href="#sync-pagination">Sync Pagination Requests</a>
    - <a href="#sync-pagination-explanation">Sync Pagination Explanation</a>
- <a href="#async-pagination">Async Pagination Requests</a>
    - <a href="#async-pagination-explanation">Async Pagination Explanation</a>
- <a href="#conclusion">Conclusion</a>
- <a href="#what-comes-next">What comes next</a>


<h2 id="intro">Intro</h2>

In the previous [async requests with SerpApi and Python](https://serpapi.com/blog/making-async-requests-with-serpapi/) blog post we've covered how to make async requests and how they work. In this continuation blog post, we'll cover how to add pagination to async requests.

📌Note: This blog post does not cover multithreading or [`asyncio`](https://docs.python.org/3/library/asyncio.html). The topic will be covered in the next blog post.

______

**Subject of test**: [YouTube Search Engine Results API](https://serpapi.com/youtube-search-api).

**Test includes**: 50 search queries, pagination per each query, and data extraction. Sync and Async used different search queries.

**Where test happend**: using [Replit](https://replit.com/)'s [Boosted](https://blog.replit.com/boosts) repls.

**What hardware used**: 4 vCPUs and 4 GB of RAM on Ubuntu 20.04.2 LTS. 

<h2 id="time-comparison">Time Comparison</h2>

The change in total elapsed time is a whopping 434% change 😵😍

Time was recorded using [`$ time python <file.py>`](https://stackoverflow.com/a/1557577/15164646):

![image](https://user-images.githubusercontent.com/78694043/201845352-e8346864-64f6-4265-9e60-f112a20afb43.png)


| Type    | Sync requests pagination | Async requests pagination | % difference     |
|---------|--------------------------|---------------------------|------------------|
| user    | 135.98s                  | 149.98s                   | -90.67% decrease |
| system  | 20.43s                   | 13.43s                    | -152.12% decrease|
| elapsed | 2h 36m 25s               | 36m 2s                    | +434.09% increase|


<h2 id="sync-pagination">Sync Pagination Requests</h2>

You can [check the code example in the online IDE](https://replit.com/@serpapi/SerpApi-YouTube-Search-Sync-with-Pagination#main.py):

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit)
import os, json

# shortened for example
queries = [
    'tent',
    'friendly',
    'ripe',
    'helpful',
    'analyze',
    'pack'
]

data = []

for query in queries:
    params = {
        'api_key': os.getenv('API_KEY'),  # serpapi api key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
    }

    search = YoutubeSearch(params)        # where data extraction happens

    while True:
        results = search.get_json()       # JSON -> Python dict

        if 'error' in results:
            print(results['error'])
            break

        for result in results.get('video_results', []):
            data.append({
                'title': result.get('title'),
                'link': result.get('link'),
                'channel': result.get('channel').get('name'),
            })
        
        if 'next' in results.get('serpapi_pagination', {}):
            search.params_dict.update(dict(parse_qsl(urlsplit(results.get('serpapi_pagination', {}).get('next')).query)))
        else:
            break

# print(json.dumps(data, indent=2, ensure_ascii=False))
```

<h2 id="sync-pagination-explanation">Sync Pagination Explanation</h2>

Import libraries:

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit) # for pagination
import os, json
```


Create a [`list`](https://www.w3schools.com/python/python_lists.asp) of search queries we want to search:

```python
queries = [
    'burly',
    'silk',
    'monkey',
    'abortive',
    'hot'
]
```

(optional) Create a temporary `list` that will store extracted data:

```python
data = []
```

Add a `for` loop to iterate over all `queries`, create [SerpApi YouTube search parameters](https://serpapi.com/youtube-search-api#api-parameters), and pass them to `YoutubeSearch` which will make a request to SerpApi. Transform the return JSON to Python `dict` via [`get_dict()`](https://github.com/serpapi/google-search-results-python/blob/56447f2ac39f202fa663233c87fa7c7b9ca1e6b2/serpapi/serp_api_client.py#L98-L103) method:

```python
for query in queries:
    params = {
        'api_key': os.getenv('API_KEY'),  # serpapi api key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
    }

    search = YoutubeSearch(params)       # where data extraction happens
    results = search.get_dict()          # JSON -> Python dict
```

[Check for `'errors'`](https://github.com/serpapi/google-search-results-python#error-management), iterate over video results and extract needed data to the temporary `list`.

```python
if 'error' in results:
    print(results['error'])
    break
```

After that, we need to check for the `'next'` key from `'serpapi_pagination'` and update search parameters data to a data from the next page, or exit the infinite loop if there are no more pages left:   

```python
for result in results.get('video_results', []):
    data.append({
        'title': result.get('title'),
        'link': result.get('link'),
        'channel': result.get('channel').get('name')
    })

if 'next' in results.get('serpapi_pagination', {}):
    search.params_dict.update(dict(parse_qsl(urlsplit(results.get('serpapi_pagination', {}).get('next')).query)))
else:
    break
```


<h2 id="async-pagination">Async Pagination Requests</h2>

You can [check the code example in the online IDE](https://replit.com/@serpapi/SerpApi-YouTube-Search-Async-with-Pagination#main.py):

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit)
from queue import Queue
import os, json

queries = [
    'object',
    'far-flung',
    'gabby',
    'tiresome',
    'scatter',
    'exclusive',
    'wealth'
]

search_queue = Queue()

for query in queries:
    params = {
        'api_key': os.getenv('API_KEY'),  # serpapi api key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
        'async': True,                    # async batch requests
    }

    search = YoutubeSearch(params)       # where data extraction happens
    results = search.get_dict()          # JSON -> Python dict
    
    if 'error' in results:
        print(results['error'])
        break

    print(f"add search to the queue with ID: {results['search_metadata']}")
    search_queue.put(results)

data = []

while not search_queue.empty():
    result = search_queue.get()
    search_id = result['search_metadata']['id']

    print(f'Get search from archive: {search_id}')
    search_archived = search.get_search_archive(search_id) # where all extracted data is stored and accessed
    
    print(f"Search ID: {search_id}, Status: {search_archived['search_metadata']['status']}")

    if re.search(r'Cached|Success', search_archived['search_metadata']['status']):
        for video_result in search_archived.get('video_results', []):
            data.append({
                'title': video_result.get('title'),
                'link': video_result.get('link'),
                'channel': video_result.get('channel').get('name'),
            })
            
        if 'next' in search_archived.get('serpapi_pagination', {}):
            search.params_dict.update(dict(parse_qsl(urlsplit(search_archived['serpapi_pagination']['next']).query)))
            
            new_page_result = search.get_dict() # new results from updated (next) page
            search_queue.put(new_page_result)   # add to queue results from updated (next) page
    else:
        print(f'Requeue search: {search_id}')
        search_queue.put(result)
        
# print(json.dumps(data, indent=2))
print('all searches completed')
```

<h2 id="async-pagination-explanation">Async Pagination Explanation</h2>

Import libraries:

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit) # for pagination
from queue import Queue                        # for async requests 
import os, json
```

Create a `list` of search queries you want to search:

```python
queries = [
    'object',
    'far-flung',
    'gabby',
    'tiresome',
    'scatter',
    'exclusive',
    'wealth'
]
```

[Create a `Queue`](https://docs.python.org/3/library/queue.html#module-queue) that will store all requests that have been sent to SerpApi for processing:

```python
search_queue = Queue()
```

Iterate over all queries, create [SerpApi YouTube search parameters](https://serpapi.com/youtube-search-api#api-parameters) with `'async': True` parameter present. Check for errors and [`put()` `search` in the queue](https://docs.python.org/3/library/queue.html#queue.Queue.put):

```python
for query in queries:
    params = {
        'api_key': os.getenv('API_KEY'),  # serpapi api key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
        'async': True,                    # async batch requests
    }

    search = YoutubeSearch(params)       # where data extraction happens
    results = search.get_dict()          # JSON -> Python dict
    
    if 'error' in results:
        print(results['error'])
        break

    print(f"add search to the queue with ID: {results['search_metadata']}")
    search_queue.put(results)
```

Create a temporary `list` that will be used to store extracted data from the search archive API:

```python
data = []
```

[Iterate through all queue until it's `empty()`](https://docs.python.org/3/library/queue.html#queue.Queue.empty) and get the data from search archive by accessing search ID:

```python
while not search_queue.empty():
    result = search_queue.get()
    search_id = result['search_metadata']['id']

    print(f'Get search from archive: {search_id}')
    search_archived = search.get_search_archive(search_id) # where all extracted data is stored and accessed
    
    print(f"Search ID: {search_id}, Status: {search_archived['search_metadata']['status']}")
```

Check if the search is either *cached* or *succeeded*, if so, extract the needed data. 

After extracting data we need to check if the `'next'` page is present inside `'serpapi_pagination'`. If it's present, we need to assign new page data to a new variable and `put()` it to the `Queue`.

If `'next'` key is not present, we need to exit the pagination and proceed to requeuing result or exit `while` loop depending on the result `'status'`:

```python
if re.search(r'Cached|Success', search_archived['search_metadata']['status']):
        for result in search_archived.get('video_results', []):
            data.append({
                'title': result.get('title'),
                'link': result.get('link'),
                'channel': result.get('channel').get('name')
            })

        if 'next' in search_archived.get('serpapi_pagination', {}):
            search.params_dict.update(dict(parse_qsl(urlsplit(search_archived['serpapi_pagination']['next']).query)))
            
            new_page_result = search.get_dict() # new results from updated (next) page
            search_queue.put(new_page_result)   # add to queue results from updated (next) page
    else:
        print(f'Requeue search: {search_id}')
        search_queue.put(result)
        
print(json.dumps(data, indent=2))
print('all searches completed')
```

<h2 id="conclusion">Conclusion</h2>

As you so In this comparison, you saw a 430% speed increase when using Async in comparison Sync requests combined with pagination.

With that said, we strongly recommend using `async` parameter if there's a need to extract data with pagination. 

<h2 id="what-comes-next">What comes next</h2>

In the next blog post we'll cover:

- how to speed up `async` requests even more.


<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a> </p>