Using Google Jobs Listing Results API from SerpApi
Tags: webscraping, tutorial, python, programming
Published: 2022-11-15 15:53:27

- <a href="#what_will_be_scraped">What will be scraped</a>
- <a href="#why_use_api">Why using API?</a>
- <a href="#fullcode">Full Code</a>
- <a href="#preparation">Preparation</a>
- <a href="#code_explanation">Code Explanation</a>
    - <a href="#top_level_code_environment">Top-level code environment</a>
    - <a href="#extract_multiple_jobs">Extract Multiple Jobs</a>
    - <a href="#scrape_google_jobs_listing">Scrape Google Jobs Listing</a>
- <a href="#output">Output</a>
- <a href="#links">Links</a>


<h2 id="what_will_be_scraped">What will be scraped</h2>

![wwbs-google-jobs-listing](https://user-images.githubusercontent.com/81998012/199747185-9d686177-c3c8-453c-ac81-bcaf546a25b1.png)

📌Note: Some queries may not display all sections. You can check your query in the [playground](https://serpapi.com/playground?engine=google_jobs_listing&q=eyJqb2JfdGl0bGUiOiJCYXJpc3RhIiwiY29tcGFueV9uYW1lIjoiQmx1ZSBCb3R0bGUgQ29mZmVlIiwiY29tcGFueV9taWQiOiIvbS8wM3AxMnFkIiwiYWRkcmVzc19jaXR5IjoiU2FuIEZyYW5jaXNjbyIsImFkZHJlc3Nfc3RhdGUiOiJDYWxpZm9ybmlhIiwiaHRpZG9jaWQiOiJBNnFRZGw1VjZvU1ZxbnI0QUFBQUFBPT0iLCJ1dWxlIjoidytDQUlRSUNJa1UyRnVJRVp5WVc1amFYTmpieUJDWVhrZ1FYSmxZU3hWYm1sMFpXUWdVM1JoZEdWeiIsImZjIjoiRW80QkNtY3dZVFZSY0RkWE9VTkJSek42WVdocVVHRkZXV2hqTkdGamJGSm5lRUpmWVhOc2NVRmlhRXhKWHpaek1Hb3hWamx2UWxKSlpGVnBVRUpyV1ZkeU1sQXpXRmhVVDB3MVRHbFNjamh5T0VzelRVcGhhM2x6WjFsQ1NuZFBWMjVJTlVKQlJWTjJOREZCRWhaME5rUXdXSE5pYjB4TmRWVXRaMVEwTkc5SFNVTkJHZ3RQTFVKSGRqaFNVWEp4TkEiLCJmY2VudiI6IjEiLCJmY3YiOiIyIiwiZmNfaWQiOiJmY183In0).


<h2 id="why_use_api">Why using API?</h2>

- No need to create a parser from scratch and maintain it.
- Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
- Pay for proxies, and CAPTCHA solvers.
- Don't need to use browser automation.

SerpApi handles everything on the backend with fast response times under ~2.5 seconds (~1.2  seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under [SerpApi Status page](https://serpapi.com/status).

![serpapi-status-page](https://user-images.githubusercontent.com/78694043/200245238-b1879700-907c-4904-a596-8712aaabfb99.png)


<h2 id="fullcode">Full Code</h2>

If you don't need explanation, have a look at [full code example in the online IDE](https://replit.com/@chukhraiartur/blog-google-jobs-listing#main.py).

```python
from serpapi import GoogleSearch
import os, json


def extract_multiple_jobs():
    params = {
        # https://docs.python.org/3/library/os.html#os.getenv
        'api_key': os.getenv('API_KEY'),        # your serpapi api
        'engine': 'google_jobs',                # SerpApi search engine	
        'gl': 'us',                             # country of the search
        'hl': 'en',                             # language of the search
        'q': 'barista new york',                # search query
    }

    search = GoogleSearch(params)               # where data extraction happens on the SerpApi backend
    results = search.get_dict()                 # JSON -> Python dict

    return [job.get('job_id') for job in results['jobs_results']]


def scrape_google_jobs_listing(job_ids):
    data = []

    for job_id in job_ids:
        params = {
            # https://docs.python.org/3/library/os.html#os.getenv
            'api_key': os.getenv('API_KEY'),    # your serpapi api
            'engine': 'google_jobs_listing',    # SerpApi search engine	
            'q': job_id,                        # search query (job_id)
        }
        
        search = GoogleSearch(params)           # where data extraction happens on the SerpApi backend
        results = search.get_dict()             # JSON -> Python dict
        
        data.append({
            'job_id': job_id,
            'apply_options': results.get('apply_options'),
            'salaries': results.get('salaries'),
            'ratings': results.get('ratings')
        })

    return data


def main():
    job_ids = extract_multiple_jobs()
    google_jobs_listing_results = scrape_google_jobs_listing(job_ids)

    print(json.dumps(google_jobs_listing_results, indent=2, ensure_ascii=False))


if __name__ == '__main__':
    main()
```


<h2 id="preparation">Preparation</h2>

Install library:

```lang-none
pip install google-search-results
```

[`google-search-results`](https://github.com/serpapi/google-search-results-python) is a SerpApi API package.


<h2 id="code_explanation">Code Explanation</h2>

Import libraries:

```python
from serpapi import GoogleSearch
import os, json
```

| Library       | Purpose                                                            |
|---------------|--------------------------------------------------------------------|
| `GoogleSearch` | to scrape and parse Google results using [SerpApi](https://serpapi.com/) web scraping library. |
| [`os`](https://docs.python.org/3/library/os.html)| to return environment variable (SerpApi API key) value.  |
| [`json`](https://docs.python.org/3/library/json.html)| to convert extracted data to a JSON object. |


<h3 id="top_level_code_environment">Top-level code environment</h3>

The `extract_multiple_jobs()` function is called to get all the `job_id` values. The resulting list of `job_ids` is passed to the `scrape_google_jobs_listing(job_ids)` function to retrieve the required data. The explanation of these functions will be in the corresponding headings below.

This code uses the generally accepted rule of using the [`__name__ == "__main__"`](https://docs.python.org/3/library/__main__.html#) construct: 

```python
def main():
    job_ids = extract_multiple_jobs()
    google_jobs_listing_results = scrape_google_jobs_listing(job_ids)

    print(json.dumps(google_jobs_listing_results, indent=2, ensure_ascii=False))


if __name__ == '__main__':
    main()
```

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video [Python Tutorial: if __name__ == '__main__'](https://www.youtube.com/watch?v=sugvnHA7ElY&t=1s) for more details.


<h3 id="extract_multiple_jobs">Extract Multiple Jobs</h3>

The function returns a list of `job_id` values. The value of this identifier will be used in the next function to create the request.

This function provides a code snippet for getting data from the first page. If you want to extract data using pagination, you can see it in the [Scrape Google Jobs organic results with Python](https://serpapi.com/blog/scrape-google-jobs-organic-results-with-python/#using-google-jobs-api-from-serpapi) blog post.

At the beginning of the function, parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the `params` dictionary. 

```python
params = {
    # https://docs.python.org/3/library/os.html#os.getenv
    'api_key': os.getenv('API_KEY'),        # your serpapi api
    'engine': 'google_jobs',                # SerpApi search engine	
    'gl': 'us',                             # country of the search
    'hl': 'en',                             # language of the search
    'q': 'barista new york',                # search query
}
```

|Parameters|Explanation|
|----------|-----------|
|[`api_key`](https://serpapi.com/google-jobs-api#api-parameters-serpapi-parameters-api-key)| Parameter defines the SerpApi private key to use.|
|[`engine`](https://serpapi.com/google-jobs-api#api-parameters-serpapi-parameters-engine)| Set parameter to `google_jobs` to use the Google Jobs API engine.|
|[`gl`](https://serpapi.com/google-jobs-api#api-parameters-localization-gl)| Parameter defines the country to use for the Google search. It's a two-letter country code. (e.g., `us` for the United States, `uk` for United Kingdom, or `fr` for France). Head to the [Google countries page](https://serpapi.com/google-countries) for a full list of supported Google countries.|
|[`hl`](https://serpapi.com/google-jobs-api#api-parameters-localization-hl)| Parameter defines the language to use for the Google Jobs search. It's a two-letter language code. (e.g., `en` for English, `es` for Spanish, or `fr` for French). Head to the [Google languages page](https://serpapi.com/google-languages) for a full list of supported Google languages.|
|[`q`](https://serpapi.com/google-jobs-api#api-parameters-search-query-q)| Parameter defines the query you want to search.|

Then, we create a `search` object where the data is retrieved from the SerpApi backend. In the `results` dictionary we get data from JSON:

```python
search = GoogleSearch(params)   # where data extraction happens on the SerpApi backend
results = search.get_dict()     # JSON -> Python dict
```

Returns a compiled list of all `job_id` using [list comprehension](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions):

```python
return [job.get('job_id') for job in results['jobs_results']]
```

The function looks like this:

```python
def extract_multiple_jobs():
    params = {
        # https://docs.python.org/3/library/os.html#os.getenv
        'api_key': os.getenv('API_KEY'),        # your serpapi api
        'engine': 'google_jobs',                # SerpApi search engine	
        'gl': 'us',                             # country of the search
        'hl': 'en',                             # language of the search
        'q': 'barista new york',                # search query
    }

    search = GoogleSearch(params)               # where data extraction happens on the SerpApi backend
    results = search.get_dict()                 # JSON -> Python dict

    return [job.get('job_id') for job in results['jobs_results']]
```


<h3 id="scrape_google_jobs_listing">Scrape Google Jobs Listing</h3>

This function takes the `job_ids` list and returns a list of all data.

Declaring the `data` list where the extracted data will be added:

```python
data = []
```

For each `job_id` value in the `job_ids` list, separate requests will be made and the corresponding data will be retrieved:

```python
for job_id in job_ids:
    # data extraction will be here
```

Next, we write a parameters for making a request:

```python
params = {
    # https://docs.python.org/3/library/os.html#os.getenv
    'api_key': os.getenv('API_KEY'),    # your serpapi api
    'engine': 'google_jobs_listing',    # SerpApi search engine	
    'q': job_id,                        # search query (job_id)
}
```

|Parameters|Explanation|
|----------|-----------|
|[`api_key`](https://serpapi.com/google-jobs-listing-api#api-parameters-serpapi-parameters-api-key)| Parameter defines the SerpApi private key to use.|
|[`engine`](https://serpapi.com/google-jobs-listing-api#api-parameters-serpapi-parameters-engine)| Set parameter to `google_jobs_listing` to use the Google Jobs Listing API engine.|
|[`q`](https://serpapi.com/google-jobs-listing-api#api-parameters-search-query-q)| Parameter defines the `job_id` string which can be obtained from [Google Jobs API](https://serpapi.com/google-jobs-api).|

Then, we create a `search` object where the data is retrieved from the SerpApi backend. In the `results` dictionary we get data from JSON:

```python
search = GoogleSearch(params)   # where data extraction happens on the SerpApi backend
results = search.get_dict()     # JSON -> Python dict
```

We can then create a dictionary structure from values such as `job_id`, `apply_options`, `salaries`, and `ratings`. The extracted data is written according to the corresponding keys. After that, the dictionary is appended to the `data` list:

```python
data.append({
    'job_id': job_id,
    'apply_options': results.get('apply_options'),
    'salaries': results.get('salaries'),
    'ratings': results.get('ratings')
})
```

At the end of the function, the `data` list is returned with the retrieved data for each `job_id`:

```python
return data
```

The complete function to scrape all data would look like this:

```python
def scrape_google_jobs_listing(job_ids):
    data = []

    for job_id in job_ids:
        params = {
            # https://docs.python.org/3/library/os.html#os.getenv
            'api_key': os.getenv('API_KEY'),    # your serpapi api
            'engine': 'google_jobs_listing',    # SerpApi search engine	
            'q': job_id,                        # search query (job_id)
        }
        
        search = GoogleSearch(params)           # where data extraction happens on the SerpApi backend
        results = search.get_dict()             # JSON -> Python dict
        
        data.append({
            'job_id': job_id,
            'apply_options': results.get('apply_options'),
            'salaries': results.get('salaries'),
            'ratings': results.get('ratings')
        })

    return data
```


<h2 id="output">Output</h2>

```json
[
  {
    "job_id": "eyJqb2JfdGl0bGUiOiJCYXJpc3RhIiwiaHRpZG9jaWQiOiJuc3Y1d1hyNXdFOEFBQUFBQUFBQUFBPT0iLCJnbCI6InVzIiwiaGwiOiJlbiIsImZjIjoiRXVJQkNxSUJRVUYwVm14aVFtcFdYMjl0V0ZadU9USTNWV0ZZUlZRek9XRTJPVlJtYUc1RVZtaGpaRk5WT1VFMlNYZFpaR2ROU0dzdFoyMVBkMmxmUTNKS2RUQnJjMWxFT0dZNFNHWnFXRUZNTjB4eFRWVmtMV1JRVVRWaVJGbFVSMVo1YmxsVWVuazVPRzlxVVVsTmVXcFJjRXhPVWpWbWMwdFlTMlo2V21SUU1XSkZZa2hTY2pKaGRYcEdlRzVxTVVWNGIwZ3lhVXd3UlZGVVZ6Tk5XSGRNYXpKbVYyVjNFaGQzYkhCeFdTMWZUMHhNTW01d2RGRlFNRGhwUW05QmF4b2lRVVJWZVVWSFpqSTJWMjF3TjBoU2FtNDRPSHB5WkVWTldVMVhVWGRTU1hwMVFRIiwiZmN2IjoiMyIsImZjX2lkIjoiZmNfMSIsImFwcGx5X2xpbmsiOnsidGl0bGUiOiIubkZnMmVie2ZvbnQtd2VpZ2h0OjUwMH0uQmk2RGRje2ZvbnQtd2VpZ2h0OjUwMH1BcHBseSBkaXJlY3RseSBvbiBDdWxpbmFyeSBBZ2VudHMiLCJsaW5rIjoiaHR0cHM6Ly9jdWxpbmFyeWFnZW50cy5jb20vam9icy80MTc4NjMtQmFyaXN0YT91dG1fY2FtcGFpZ249Z29vZ2xlX2pvYnNfYXBwbHlcdTAwMjZ1dG1fc291cmNlPWdvb2dsZV9qb2JzX2FwcGx5XHUwMDI2dXRtX21lZGl1bT1vcmdhbmljIn19",
    "apply_options": [
      {
        "title": "Apply on Trabajo.org",
        "link": "https://us.trabajo.org/job-1683-20221107-34e191c4eb8c8ca3ec69adfa55061df2?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
      },
      {
        "title": "Apply on Jobs",
        "link": "https://us.fidanto.com/jobs/job-opening/nov-2022/barista-1432712052?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
      },
      {
        "title": "Apply on Craigslist",
        "link": "https://newyork.craigslist.org/mnh/fbh/d/new-york-cafe-barista/7553733276.html?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
      },
      {
        "title": "Apply directly on Culinary Agents",
        "link": "https://culinaryagents.com/jobs/417863-Barista?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
      }
    ],
    "salaries": null,
    "ratings": null
  },
  ... other results
]
```


<h2 id="links">Links</h2>

- [Code in the online IDE](https://replit.com/@chukhraiartur/blog-google-jobs-listing#main.py)
- [Google Jobs Listing Results API](https://serpapi.com/google-jobs-listing-api)
- [Google Jobs API](https://serpapi.com/google-jobs-api)

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>

<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>