How to Extract Bing News Data with SerpApi and Python
Tags: webscraping, python, tutorial, programming
Published: 2023-03-29 04:00:00

- <a href="#intro">Intro</a>
- <a href="#what_will_be_scraped">What will be scraped</a>
- <a href="#why_use_api">Why using API?</a>
- <a href="#fullcode">Full Code</a>
- <a href="#preparation">Preparation</a>
- <a href="#code_explanation">Code Explanation</a>
- <a href="#output">Output</a>
- <a href="#links">Links</a>


<h2 id="intro">Intro</h2>

In this blog post, we'll go through the process of extracting Bing News using the [Bing News Engine Results API](https://serpapi.com/bing-news-api) and the Python programming language. You can [look at the complete code in the online IDE (Replit)](https://replit.com/@chukhraiartur/blog-bing-news-api#main.py).

<h2 id="what_will_be_scraped">What will be scraped</h2>

![wwbs-bing-news](https://user-images.githubusercontent.com/81998012/228002802-995e1d57-d05c-4051-9354-4731e5828cbd.png)


<h2 id="why_use_api">Why using API?</h2>

There're a couple of reasons that may use API, ours in particular:

- No need to create a parser from scratch and maintain it.
- Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
- Pay for proxies, and CAPTCHA solvers.
- Figure out the legal part of scraping data.

SerpApi handles everything on the backend with fast response times under ~1.5 seconds (~1.0  seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under [SerpApi Status page](https://serpapi.com/status).

![serpapi-status-all](https://user-images.githubusercontent.com/81998012/228003161-662d724c-ef7c-4385-96c1-091bcbf02a57.png)


<h2 id="fullcode">Full Code</h2>

This code retrieves all news with pagination:

```python
from serpapi import BingSearch
import json

params = {
    'api_key': '...',                   # https://serpapi.com/manage-api-key
    'q': 'Coffee',                      # search query
    'engine': 'bing_news',              # search engine
    'cc': 'US',                         # country of the search
    'first': 1,                         # pagination
    'count': 10,                        # number of results per page
    'qft': 'interval="7"'               # news for past 24 hours
}

search = BingSearch(params)             # data extraction on the SerpApi backend
results = search.get_dict()             # JSON -> Python dict

bing_news_results = []

page_count = 0
page_limit = 5

while 'error' not in results and page_count < page_limit:
    bing_news_results.extend(results.get('organic_results', []))

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()

print(json.dumps(bing_news_results, indent=2, ensure_ascii=False))
```


<h2 id="preparation">Preparation</h2>

Install library:

```lang-none
pip install google-search-results
```

[`google-search-results`](https://github.com/serpapi/google-search-results-python) is a SerpApi API package.


<h2 id="code_explanation">Code Explanation</h2>

Import libraries:

```python
from serpapi import BingSearch
import json
```

| Library       | Purpose                                                            |
|---------------|--------------------------------------------------------------------|
| `BingSearch` | to scrape and parse Bing results using [SerpApi](https://serpapi.com/) web scraping library. |
| [`json`](https://docs.python.org/3/library/json.html)| to convert extracted data to a JSON object. |


The parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the `params` dictionary:

```python
params = {
    'api_key': '...',                   # https://serpapi.com/manage-api-key
    'q': 'Coffee',                      # search query
    'engine': 'bing_news',              # search engine
    'cc': 'US',                         # country of the search
    'first': 1,                         # pagination
    'count': 10,                        # number of results per page
    'qft': 'interval="7"'               # news for past 24 hours
}
```

|Parameters|Explanation|
|----------|-----------|
|[`api_key`](https://serpapi.com/bing-news-api#api-parameters-serpapi-parameters-api-key)| Parameter defines the SerpApi private key to use.|
|[`q`](https://serpapi.com/bing-news-api#api-parameters-search-query-q)| Parameter defines the search query. You can use anything that you would use in a regular Bing search. (e.g., `'query'`, `NOT`, `OR`, `site:`, `filetype:`, `near:`, `ip:`, `loc:`, `feed:` etc.).|
|[`engine`](https://serpapi.com/bing-news-api#api-parameters-serpapi-parameters-engine)| Set parameter to `bing_news` to use the Bing News API engine.|
|[`cc`](https://serpapi.com/bing-news-api#api-parameters-localization-cc)| Parameter defines the country to search from. It follows the 2-character [ISO_3166-1](https://en.wikipedia.org/wiki/ISO_3166-1) format. (e.g., `us` for United States, `de` for Germany, `gb` for United Kingdom, etc.).|
|[`first`](https://serpapi.com/bing-news-api#api-parameters-pagination-first)| Parameter controls the offset of the organic results. This parameter defaults to `1`. (e.g., `first=10` will move the 10th organic result to the first position).|
|[`count`](https://serpapi.com/bing-news-api#api-parameters-pagination-count)| Parameter controls the number of results per page. This parameter is only a suggestion and might not reflect actual results returned.|
|[`qft`](https://serpapi.com/bing-news-api#api-parameters-advanced-bing-news-parameters-qft)| Parameter defines results sorted by date. If the parameter is not set, it will default to the "Best match" sorting. It can be set to: `interval="4"` - Past hour, `interval="7"` - Past 24 hours, `interval="8"` - Past 7 days, `interval="9"` - Past 30 days, `sortbydate="1"` - Most Recent.|

📌Note: You can also add other [API Parameters](https://serpapi.com/bing-news-api#api-parameters).

Then, we create a `search` object where the data is retrieved from the SerpApi backend. In the `results` dictionary we get data from JSON:

```python
search = BingSearch(params)         # data extraction on the SerpApi backend
results = search.get_dict()         # JSON -> Python dict
```

Before extracting data, the `bing_news_results` list is created where this data will be added later:

```python
bing_news_results = []
```

The `page_limit` variable defines the page limit. If you want to extract data from a different number of pages, then simply write the required number into this variable.

```python
page_limit = 5
```

To get all results, you need to apply pagination. This is achieved by the following check: while there is no `error` in the `results` and the current `page_count` value is less than the specified `page_limit` value, we extract the data, increase the `first` parameter by the value of the `count` parameter to get the results from next page and update the `results` object with the new page data:

```python
page_count = 0

while 'error' not in results and page_count < page_limit:
    # data extraction from current page will be here

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()
```

Extending the `bing_news_results` list with new data from each page:

```python
bing_news_results.extend(results.get('organic_results', []))
# title = results['organic_results'][0]['title']
# link = results['organic_results'][0]['link']
# snippet = results['organic_results'][0]['snippet']
# source = results['organic_results'][0]['source']
# date = results['organic_results'][0]['date']
# thumbnail= results['organic_results'][0]['thumbnail']
```

📌Note: In the comments above, I showed how to extract specific fields. You may have noticed the `results['organic_results'][0]`. This is the index of a organic result, which means that we are extracting data from the first organic result. The `results['organic_results'][1]` is from the second organic result and so on.

After the all data is retrieved, it is output in JSON format:

```python
print(json.dumps(bing_news_results, indent=2, ensure_ascii=False))
```


<h2 id="output">Output</h2>

```json
[
  {
    "title": "Coffee drinkers get more steps but also less sleep, study finds",
    "link": "https://www.cbsnews.com/sacramento/news/coffee-drinkers-get-more-steps-but-also-less-sleep-study-finds/",
    "snippet": "Coffee is one of the most consumed beverages worldwide, but the pendulum has swung back and forth about its benefits and ...",
    "source": "CBS News",
    "date": "5h",
    "thumbnail": "https://serpapi.com/searches/6421c51af26ac67a4930a857/images/85e52d2243238795454091b5f6b3f41e7f9969cd089f02e76ad21147c47e4f69.jpeg"
  },
  {
    "title": "Cedarburg School Board candidate accused of sexually harassing young workers at his former coffee shop",
    "link": "https://www.jsonline.com/story/news/education/2023/03/27/cedarburg-school-board-candidate-accused-of-sexual-harassment-coffee-shop/70039331007/",
    "snippet": "Former employees accused Scott Sidney, a Cedarburg School Board candidate, of inappropriate words and actions at his former ...",
    "source": "Milwaukee Journal Sentinel",
    "date": "5h",
    "thumbnail": "https://serpapi.com/searches/6421c51af26ac67a4930a857/images/85e52d2243238795454091b5f6b3f41e8897032186a0a3603444deba3cdd76a7.jpeg"
  },
  {
    "title": "New development near WoodTrust in Grand Rapids; new merch from local coffee shop | Streetwise",
    "link": "https://news.yahoo.com/development-near-woodtrust-grand-rapids-100428122.html",
    "snippet": "If you know of a new business, a development or a place that’s closing, send me a note at cshuda@gannett.com. If you have ...",
    "source": "YAHOO!News",
    "date": "6h",
    "thumbnail": "https://serpapi.com/searches/6421c51af26ac67a4930a857/images/85e52d2243238795454091b5f6b3f41e5f1483416a23fbf49adc15101a042c82.jpeg"
  },
  ... other news
]
```

📌Note: Head to the [playground](https://serpapi.com/playground?engine=bing_news) for a live and interactive demo.


<h2 id="links">Links</h2>

- [Code in the online IDE](https://replit.com/@chukhraiartur/blog-bing-news-api#main.py)
- [Bing News Engine Results API](https://serpapi.com/bing-news-api)

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>

<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>