Day 75 Of 100DaysOfCode: Scrapping Different Categories Of News Of News Portal Using BeautifulSoup
Tags: womenintech, 100daysofcode, beginners, codenewbie
Published: 2021-03-13 15:52:50

Today is my 75th day of #100Daysofcode and #python learning. Today
also used the time to learn from datacamp regarding the topic pandas and also learn basic of R programming language. 
 
And most of the time I gave to my first project. Yesterday I scrapped news of only the news category. Today I am able to scrapped news of different categories like sports, business, world. And I used pandas to visualize data obtained from different categories. I am doing my project using the drive category. As being from a non-technical field saving CSV files to drive is new for me. I learned to save the CSV file to the drive. While doing so we need to mount our goggle drive
Below is my updated code for today.
###Scrapping Different Categories Of
News
```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import re
from bs4 import BeautifulSoup as BS
import requests
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
```
```python
categories={"news":"https://ekantipur.com/news/",
            "business":"https://ekantipur.com/business/",
            "world":"https://ekantipur.com/world/",
            "sports":"https://ekantipur.com/sports"
```
```python
http = urllib3.PoolManager()
http.addheaders = [('User-agent', 'Mozilla/61.0')]

news_values=[]
ndict = {'Title': [], "URL": [], "Date":[],
      "Author":[], "Author URL":[], "Content":[],"Category": []}
show=False
for category, url in categories.items():
  web_page = http.request('GET', url)
  soup = BS(web_page.data, 'html5lib')

  for title in soup.findAll("h2"):
    if title.a:
      title_link=title.a.get("href")
      # print(title_link)
      if title_link.split(":")[0]!="https":
        title_link=url.split(f"/{category}")[0]+title.a.get("href")
      title_text=title.text
      #print(title_link)
      
      news_page = http.request('GET', title_link)
      news_soup = BS(news_page.data, 'html5lib')

      date = news_soup.find("time").text
      author_url = news_soup.select_one(".author").a.get("href")
      author_name = news_soup.select_one(".author").text

      for row in news_soup.select(".row"):
        for content in row.contents:
          if content.select(".normal"):
            content=content.p.text
            break
        break
      
      catagory = url.split('/')[-1]

      ndict["Title"].append(title_text)
      ndict["URL"].append(title_link)
      ndict["Date"].append(date)
      ndict["Author"].append(author_name)
      ndict["Author URL"].append(author_url)
      ndict["Content"].append(content)
      ndict["Category"].append(category) 
      if show:
        print(f"""
                Title: {title_text}, URL: {title_link}
                Date: {date}, Author: {author_name},Category : {category},
                Author URL: {author_url},
                Content: {content}
                        """)
    # news_values.append()
```
Dataframe of above data is,
```python
df = pd.DataFrame(ndict, columns=list(ndict.keys()))
df
```
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Day 75 Of <a href="https://twitter.com/hashtag/100DaysOfCode?src=hash&amp;ref_src=twsrc%5Etfw">#100DaysOfCode</a> and <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> <br>Scrapping different categories of<br>news of news portal using BeautifulSoup<a href="https://twitter.com/hashtag/womenintech?src=hash&amp;ref_src=twsrc%5Etfw">#womenintech</a> <a href="https://twitter.com/hashtag/100DaysOfCode?src=hash&amp;ref_src=twsrc%5Etfw">#100DaysOfCode</a> <a href="https://twitter.com/hashtag/CodeNewbie?src=hash&amp;ref_src=twsrc%5Etfw">#CodeNewbie</a> <a href="https://twitter.com/hashtag/DEVCommunity?src=hash&amp;ref_src=twsrc%5Etfw">#DEVCommunity</a> <a href="https://t.co/6yKBR4YsnW">pic.twitter.com/6yKBR4YsnW</a></p>&mdash; Durga Pokharel (@mathdurga) <a href="https://twitter.com/mathdurga/status/1370759375388741633?ref_src=twsrc%5Etfw">March 13, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>