Rss-Feed 1.3
Rss news scrapper
Requires Python: >=3.6.0

# Welcome to AINEWS-BEATS
	
AINEWS-BEATS is a user inteface to give insight about recent developments and AI related products in all the sectors (Finance, Healthcare, Defence, Logistics etc). It will help 
users to identify and utilize a product related to their sector. This will first show the graphical represention of product and it's relation graphically. It also has the provision
of dispalying the whole news article.

## Libraries used

[![Python 3.6](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-brightgreen)](https://pypi.org/project/Rss-Feed/) [![RSS-Feed](https://img.shields.io/badge/RSS--Feed-v1.0-yellowgreen)](https://pypi.org/project/Rss-Feed/) [![pypi package](https://img.shields.io/badge/pypi%20package-v%200.2.0-orange)](https://pypi.org/project/Rss-Feed/) [![BeautifulSoup4](https://img.shields.io/badge/beautifulsoup4-v4.9.1-blue)](https://www.crummy.com/software/BeautifulSoup/)

#### Python
RSS-Feed package runs using Python, one of the most frequently used Programming languages. <br/><br/>Beginners Guide for [downloading Python](https://wiki.python.org/moin/BeginnersGuide/Download) <br/><br/> Tutorial to install [Python3](https://realpython.com/installing-python/)


#### RSS-Feed
RSS-Feed is a [Python3](https://en.wikipedia.org/wiki/Python_%28programming_language%29) library for scraping news articles from webpage. It can extract main body text and comments while preserving the page structure. It is packaged as a software library available from the package repository PyPi.

#### BeautifulSoup4
Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree.

## Installation
<pre>```
pip install Rss-Feed
```</pre>



## Data Collection

**Data Collection** is done by taking all the RSS-Feed link initially and then keeping it in a text file. The same text file is passed to the package RSS-Feed which will then extract.
 
 - Title of the particular article
 - Hyper-Link to the article
 - Entire Description for the news article
 - Published Date for the article

 
## Basics
<pre>```
from Rss_Feed import RssScrapper
```</pre>
<pre>```
object = RssScrapper()
object.get_all(file)
```</pre>
<br/>
This function will take the .txt file which contains all the rss feed urls as input and then it will extract the news data from each urls and then it will be stored in a csv file name.

 - param file - This is the text file in which all the rss feed urls are stored.
 - return - Return all the content of the article and stores in a dictionary of lists.
 
All the data returned will then be stored in a csv file.<br/>

This project is under active development, please make sure you keep it up-to-date to benefit from latest improvements:
<pre>```
pip install -U Rss_Feed
```</pre>


## Workflow

#### Extracting URL's from file
![Screenshot](img/ExtractURLFromTextfile.png)  <br/>
	
#### Create List of News Articles
![Screenshot](img/CreateListOfNewsArticles.png)

#### Extract Date
![Screenshot](img/Extract Date.png)

#### Extract Title
![Screenshot](img/Extract Title.png)

#### Extract URL
![Screenshot](img/URLExtract.png)

#### Extract Description
![Screenshot](img/Extract Description.png)

#### Extract Source
![Screenshot](img/Extract Source.png)

#### Convert Extracted News to Dataframe
![Screenshot](img/ConvertExtractedNewstoDataFrame.png)

## Core Functions

### Data Collection

##### Read Data
This function is used to read all the urls from the text file and it returns a list after reading.
<pre>```
def __read_data(self, file) -> list:
	self.file = file
	try:
		f1 = open(self.file, 'r')
		lines = f1.readlines()
		assert type(lines) == list, "Something wrong in reading"
		assert len(lines) != 0, "The file is empty nothing to read"
		self.success_logger.info("File read Successfully")
		return lines
	except AssertionError as e:
		self.failure_logger.error("Check the file contents")
		self.failure_logger.exception(e)
	except Exception as e:
		self.failure_logger.exception(e)		
```</pre>


##### Create Soup
This function will take the url and try to open it and convert the webpage to BeautifulSoup object.
<pre>```
def __create_soup(self, url):
	try:
		time.sleep(2)
		container = Request(url, headers=HEADERS)
		website = urlopen(container, timeout=60).read()
		self.soup = BeautifulSoup(website, 'xml')
		return self.soup
	except HTTPError as err:
		if err.code == 404:
			self.failure_logger.error("{} ----> Url doesn't exist".format(url))
			return err.code
		if err.code == 400:
			self.failure_logger.error('URL {}: ----> 400 Bad Request error'.format(url))
			return err.code
	except TimeoutError as te:
		self.failure_logger.error('URL {}: ----> Timeout error'.format(url))
		return te
	except (socket.timeout, socket.gaierror) as err:
		self.failure_logger.error('URL {}: ----> socket timeout error'.format(url))
		return err
	except Exception as e:
		self.failure_logger.error('URL {}:----> Some other error'.format(url))
		return e	
```</pre>


### Extraction

##### Get news items
This function will take bs4.BeautifulSoup object and from that it will extract all the item and entry tag, that contains the info which is to be extracted, if no element is found then it will return the empty list
<pre>```
def __items(self, soup):
	self.soup = soup
	try:
		if self.soup.findAll('item'):
			news_data = self.soup.findAll('item')
			return news_data
		elif self.soup.findAll('entry'):
			news_data = self.soup.findAll('entry')
			return news_data
		else:
			news_data = []
			return news_data
	except Exception as e:
		self.failure_logger.exception(e)
```</pre>


##### Get Data
This function will take the list which contains all item and entry tag data, from which one by one the data will be extracted and stored in the another list named as NEWS.
<pre>```
def __get_data(self, data, source):
	count = 0
	try:
		for i in data:
			self.rss_info = dict()
				# Titles
			try:
				self.rss_info['Title'] = i.title.text
			except Exception:
				self.rss_info['Title'] = None
				# URL's
			try:
				if i.find('link', {'rel': 'alternate'}):
					self.rss_info['Url'] = i.link['href']
				elif i.link:
					self.rss_info['Url'] = i.link.text
			except Exception:
				self.rss_info['Url'] = None
			# Descriptions
			try:
				pattern = re.compile('<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});')
				if i.description:
					desc = i.description.text
					desc = re.sub(pattern, '', desc)
					self.rss_info['Description'] = unicodedata.normalize("NFC", desc).encode('ascii',
																							 'ignore').decode(
						'utf8').strip('[]')
				elif i.find('content', {'type': 'html'}):
					desc = i.find('content', {'type': 'html'}).text
					desc = re.sub(pattern, '', desc)
					self.rss_info['Description'] = unicodedata.normalize("NFC", desc).encode('ascii',
																							 'ignore').decode(
						'utf8').strip('[]')
				elif i.find('content:encoded'):
					desc = i.find('content:encoded').text
					desc = re.sub(pattern, '', desc)
					self.rss_info['Description'] = unicodedata.normalize("NFC", desc).encode('ascii',
																							 'ignore').decode(
						'utf8').strip('[]')
				url = self.rss_info['Url'].strip()
				try:
					if not validators.url(url):
						self.failure_logger.error(
							"{} ----> Is not a valid URL".format(url))  # Check if url is in proper format or not
						continue
					context = self.fetch_url(url)
					if context is not None:
						desc = self.extract(context)
						if desc is not None:
							self.rss_info['Full_Description'] = desc
							count += 1  # test
						else:
							self.rss_info['Full_Description'] = None
					else:
						self.rss_info['Full_Description'] = None
				except Exception as e:
					self.rss_info['Full_Description'] = None
					self.failure_logger.error("{} ----> Not able to get Full description".format(url))
					self.failure_logger.exception(e)
			except Exception:
				self.rss_info['Description'] = None
			# Date
			try:
				if i.pubDate:
					self.rss_info['Date'] = i.pubDate.text
				elif i.published:
					self.rss_info['Date'] = i.published.text
			except Exception:
				self.rss_info['Date'] = None
			try:
				self.rss_info["Source"] = source
			except Exception:
				self.rss_info["Source"] = None
			self.News.append(self.rss_info)
		self.success_logger.info("{} ----> Completed".format(source) + " (Found {} news)".format(len(data)))
		self.success_logger.info(
			"{} ----> Completed".format(source) + " (Found {} Complete description)".format(count))
	except Exception as e:
		self.failure_logger.error("{} ----> error while getting the info".format(source))
		self.failure_logger.exception(e)
```</pre> 

 
##### Extract complete news content
This function is the main function which extract the complete content from news article.
<pre>```
def extract(self, filecontent, include_tables=True, include_comments=True, record_id=None, no_fallback=False, url=None):
	try:		
		tree = self.load_html(filecontent)
		if tree is None:
			return None
		if no_fallback is False:
			backup_tree = deepcopy(tree)
		else:
			backup_tree = None
		docmeta = dict.fromkeys(['title', 'author', 'url', 'description', 'sitename', 'date', 'categories', 'tags'])
		docmeta['id'] = record_id
		cleaned_tree = manual_cleaning(tree, include_tables)
		cleaned_tree = prune_html(cleaned_tree)
		cleaned_tree = HTML_CLEANER.clean_html(cleaned_tree)
		cleaned_tree = convert_tags(cleaned_tree)
		etree.strip_tags(cleaned_tree, 'hi')
		if include_comments is True:
			commentsbody, temp_comments, len_comments, cleaned_tree = extract_comments(cleaned_tree)
		else:
			commentsbody, temp_comments, len_comments = None, '', 0
		postbody, temp_text, len_text, sure_thing = extract_content(cleaned_tree, include_tables)
		if no_fallback is False:
			postbody, temp_text, len_text = compare_extraction(tree, backup_tree, url, postbody, temp_text, len_text)
		returnstring = xmltotxt(build_xml_output(postbody, commentsbody))
		return returnstring
	except Exception as e:
		self.failure_logger.error("{} ----> error while getting the info".format(source))
		self.failure_logger.exception(e)
```</pre>

##### Extract comments
This function will extract the comments from the news article.
<pre>```
def extract_comments(tree):
	try:
		comments_body = etree.Element('body')
		potential_tags = set(TAG_CATALOG)
		for expr in COMMENTS_XPATH:  # select tree if the expression has been found
			subtree = tree.xpath(expr)
			if not subtree:
				continue
			subtree = subtree[0]
			subtree = discard_unwanted_comments(subtree)
			etree.strip_tags(subtree, 'a', 'link', 'span')
			processed_elems = [process_comments_node(elem, potential_tags) for elem in subtree.xpath('.//*')]
			comments_body.extend(list(filter(None.__ne__, processed_elems)))
			if len(comments_body) > 0:  # if it has children
				subtree.getparent().remove(subtree)
				break
		temp_comments = trim(' '.join(comments_body.itertext()))
		return comments_body, temp_comments, len(temp_comments), tree
	except Exception as e:
		self.failure_logger.error("{} ----> error while getting the info".format(source))
		self.failure_logger.exception(e)
```</pre>

##### Extract news body
This function find the main content of a page using a set of XPath expressions, then extract relevant elements and strip them of unwanted subparts
<pre>```
def extract_content(tree, include_tables=False):
	try:
		sure_thing = False
		result_body = etree.Element('body')
		for expr in BODY_XPATH:  # iterating through different XPATHS
			subtree = tree.xpath(expr)
			if not subtree:
				continue
			subtree = subtree[0]
			subtree = discard_unwanted(subtree)
			for elem in subtree.iter('list'):
				if link_density_test(elem) is True:
					elem.getparent().remove(elem)
				else:
					elem.attrib.clear()
			if len(subtree) == 0:	# Skip if empty tree
				continue
			etree.strip_tags(subtree, 'a', 'link', 'span')
			potential_tags = set(TAG_CATALOG)
			if include_tables is True:
				potential_tags.add('table')
			if not subtree.xpath('//p//text()'):
				potential_tags.add('div')
			result_body.extend([e for e in
								[handle_textelem(e, potential_tags) for e in subtree.xpath('.//*')]
								if e is not None])
			try:
				while result_body[-1].tag == 'head' and result_body[-1].tail is None:
					result_body[-1].getparent().remove(result_body[-1])
			except IndexError:
				continue
			if len(result_body) > 0:	# Exit the loop if the result has children
				sure_thing = True
				break
		temp_text = trim(' '.join(result_body.itertext()))
		len_text = len(temp_text)
		if len(result_body) == 0 or len_text < MIN_EXTRACTED_SIZE:
			result_body = recover_wild_paragraphs(tree, result_body)
			temp_text = trim(' '.join(result_body.itertext()))
			len_text = len(temp_text)
		etree.strip_elements(result_body, 'done')
		etree.strip_tags(result_body, 'div')
		return result_body, temp_text, len_text, sure_thing
	except Exception as e:
		self.failure_logger.error("{} ----> error while getting the info".format(source))
		self.failure_logger.exception(e)
```</pre>


<br/>

### Pre-Processing
##### Building xml output
Building XML output based on the extracted news body and comments body.
<pre>```
def build_xml_output(postbody, commentsbody):
    output = etree.Element('doc')
    postbody.tag = 'main'
    output.append(postbody)
    if commentsbody is not None:
        commentsbody.tag = 'comments'
        output.append(commentsbody)
    return output
```</pre>

##### XML to TXT
Converting the XML output to plain text format
<pre>```
def xmltotxt(xmloutput):
    returnlist = []
    for element in xmloutput.xpath('//hi|//link'):
        merge_with_parent(element)
        continue
    for element in xmloutput.iter():
        if element.text is None and element.tail is None:
            if element.tag in ('row', 'table'):
                returnlist.append('\n')
            continue
        textelement = replace_element_text(element)
        if element.tag in ('code', 'fw', 'head', 'lb', 'list', 'p', 'quote', 'row', 'table'):
            returnlist.extend(['\n', textelement, '\n'])
        elif element.tag == 'item':
            returnlist.extend(['\n- ', textelement, '\n'])
        elif element.tag == 'cell':
            returnlist.extend(['|', textelement, '|'])
        elif element.tag == 'comments':
            returnlist.append('\n\n')
        else:
            returnlist.extend([textelement, ' '])
    return sanitize(''.join(returnlist))
```</pre>


### Helpers

##### Load HTML
This function loads the object given as Input and validate its type.
<pre>```
def load_html(self, htmlobject):
	if isinstance(htmlobject, (etree._ElementTree, html.HtmlElement)):
		return htmlobject
	tree = None
	check_flag = False
			if isinstance(htmlobject, bytes):  # try to detect encoding and convert to string
		if 'html' not in htmlobject[:50].decode(encoding='ascii', errors='ignore'):
			check_flag = True
		guessed_encoding = self.detect_encoding(htmlobject)
		if guessed_encoding is not None:
			if guessed_encoding == 'UTF-8':
				tree = html.fromstring(htmlobject, parser=self.HTML_PARSER)
			else:
				try:
					htmlobject = htmlobject.decode(guessed_encoding)
					tree = html.fromstring(htmlobject, parser=self.HTML_PARSER)
				except UnicodeDecodeError:
					self.failure_logger.error('encoding issue: %s', guessed_encoding)
					tree = html.fromstring(htmlobject, parser=self.RECOVERY_PARSER)
		else:
			tree = html.fromstring(htmlobject, parser=self.RECOVERY_PARSER)
	elif isinstance(htmlobject, str):
		if 'html' not in htmlobject[:50]:
			check_flag = True
		try:
			tree = html.fromstring(htmlobject, parser=self.HTML_PARSER)
		except ValueError:
			try:
				tree = html.fromstring(htmlobject.encode('utf8'), parser=self.HTML_PARSER)
			except Exception as err:
				self.failure_logger.error('parser bytestring %s', err)
		except Exception as err:
			self.failure_logger.error('parsing failed: %s', err)
	else:
		self.failure_logger.error('this type cannot be processed: %s', type(htmlobject))
	if tree is not None and check_flag is True:
		if len(tree) < 2:
			self.failure_logger.error('Parse tree empty: not valid HTML')
			tree = None
	return tree
```</pre>


##### Convert Tags
Simplifying and converting HTML tags to XML standards.
<pre>```
def convert_tags(tree):
    etree.strip_tags(tree, MANUALLY_STRIPPED)
    for elem in tree.iter('ul', 'ol', 'dl'):
        elem.tag = 'list'
        for subelem in elem.iter('dd', 'dt', 'li'):
            subelem.tag = 'item'
        for subelem in elem.iter('a'):
            subelem.tag = 'link'
    etree.strip_tags(tree, 'a')		# delete links
    for elem in tree.iter('h1', 'h2', 'h3', 'h4', 'h5', 'h6'):		# head tags + delete attributes
        elem.tag = 'head'
        elem.attrib.clear()
    for elem in tree.iter('br', 'hr'):		# br â†’ lb
        elem.tag = 'lb'
        elem.attrib.clear()
    for elem in tree.iter('blockquote', 'pre', 'q'):
        elem.tag = 'quote'
        elem.attrib.clear()
    for elem in tree.iter('em', 'i'):    # italics
        elem.attrib.clear()
        elem.tag = 'hi'
        elem.set('rend', '#i')
    for elem in tree.iter('b', 'strong'):    # bold font
        elem.attrib.clear()
        elem.tag = 'hi'
        elem.set('rend', '#b')
    for elem in tree.iter('u'):    # u (very rare)
        elem.tag = 'hi'
        elem.set('rend', '#u')
    for elem in tree.iter('tt'):    # tt (very rare)
        elem.attrib.clear()
        elem.tag = 'hi'
        elem.set('rend', '#t')
    for elem in tree.iter('sub'):    # sub and sup (very rare)
        elem.attrib.clear()
        elem.tag = 'hi'
        elem.set('rend', '#sub')
    for elem in tree.iter('sup'):
        elem.attrib.clear()
        elem.tag = 'hi'
        elem.set('rend', '#sup')
    for elem in tree.iter('del', 's', 'strike'):
        elem.attrib.clear()
        elem.tag = 'del'
        elem.set('rend', 'overstrike')
    return tree
```</pre>
 
##### Manual Cleaning
Pruning the tree by discarding unwanted elements.
<pre>```
def manual_cleaning(tree, include_tables):
    if include_tables is False:
        MANUALLY_CLEANED.append('table')
    for expression in MANUALLY_CLEANED:		# iterating through expression in MANUALLY_CLEANED
        for element in tree.getiterator(expression):
            element.drop_tree()
    return tree
```</pre>

##### Data Cleaning
Cleaning the news content received from each URL.
<pre>```
def cleaning(self, description):
	if type(description) == str:
		description = re.sub(pattern, '', description)
		description = unicodedata.normalize("NFC", description).encode('ascii', 'ignore').decode('utf8').strip('[]')
		final_desc = ""
		for char in description:
			if char not in punctuations:
				final_desc = final_desc + char
		final_desc = final_desc.replace("\n", " ")
		return final_desc
	else:
		return None
```</pre>

##### Domain Names
Function to map URL's with the domains. The URL to domain mapping is present in the "url_to_domain" dictionary
<pre>```
def get_domain(self, link):
	if link in url_to_domain:
		return url_to_domain[link]
	else:
		return None
```</pre>

##### Sanitize
Convert text and discard incompatible and invalid characters
<pre>```
def sanitize(text):
    try:
        return '\n'.join([l for l in (line_processing(l) for l in text.splitlines()) if l is not None])
    except AttributeError:
        return None
```</pre>

## Handling Exception Scenarios

| Exception   | How Exceptions are Handled|
| ------------   | :-------------: |
| No argument passed | Exception raised as with message as 'File Not Passed' in log file. |
| Wrong data source from user | Exception raised with appropriate error message in log file.  |
| Improper data / URL in the text file | Exception raised in log file with appropriate error message.  |
| HTTP Bad Request | Exception raised for HTTPError in the log file with appropriate error message. |
| Timeout | Exception raised for TimeoutError in the log file with appropriate error message. |
| Socket Timeout | Exception raised for Socket timeout in the log file with appropriate error message. |
| Tags Not Present | Exception raised with appropriate error message in log file. |
| Encoding issue | Appropriate error message raised in log file. |
| Other Exceptions | Exception raised in the log file with appropriate error message. |