Don't blindly remove STOPWORDS for a Sentiment Analysis Model
Tags: machinelearning, datascience, tutorial
Published: 2020-07-26 02:04:20

<h3><b>Does removing stopwords really improve model performance?</b></h3>

Hey Peeps!!
    Before creating any model, data preprocessing is must
Data preprocessing includes Data Cleaning, Data Transformation and Data Reduction

<h4>Data Cleaning:</h4> It involves handling of missing data, noisy data etc..
<ul>
<li>Missing Data</li>
<li>Noisy Data</li>
</ul>

<h4>Data Transformation:</h4> This step is taken in order to transform the data in appropriate forms suitable for the mining process
<ul>
<li>Normalization</li>
</ul>

<h4>Data Reduction:</h4> While working with a huge volume of data, analysis became harder in such cases. In order to get rid of this, we use data reduction techniques.
<ul>
<li>Dimensionality Reduction</li>
</ul>


And I started working on Amazon Fine Food Review where I got dataset from <a href="https://www.kaggle.com/snap/amazon-fine-food-reviews">Kaggle</a>.
The Main Objective of this model is to determine whether the review is positive or negative 

And I started data preprocessing before training the dataset

<h4>Steps in preprocessing:</h4>
<ul>
<li>Begin by removing the html tags.</li>
<li>Remove any punctuations or limited set of special characters like , or . etc.</li>
<li>Check if the word is made up of english letters and is not alpha-numeric</li>
<li>Check to see if the length of the word is greater than 2 (as it was researched that there is no adjective in 2-letters)</li>
<li>Convert the word to lowercase</li>
<li><b>Remove Stopwords</b></li>
<li>Finally Snowball Stemming the word</li>
</ul>

And I used Naive Bayes algorithm to train my dataset and tested it, unfortunately, my model is underperforming.
After reviewing the model, I came to know that it is because of <b>removing StopWords</b>, yes you heard it right

The most common method to remove stopwords is using NLTK's stopwords.

```python
import nltk
nltk.download('stopwords')
stop_words = set(stopwords.words('english'))
sno = nltk.stem.SnowballStemmer('english')
print(stop_words)
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/vrqhwkz5mtsemzowg4aa.PNG)

The main objective of building this model is to determine the given review is positive or negative but performing stopwords it removes the negative words which indeed it literally changes the whole meaning of the review i.e negative to postive
Ex:
<table>
<tr><th>Before Stopwords</th><th>After Stopwords</th></tr>
<tr><td>The product is really very good <b>(Positive)</b></td><td>product really good<b>(Positive)</b></td>
<tr><td>The products seems to be good.<b>(Positive)</b></td><td>products seems good <b>(Positive)</b></td></tr>
<tr><td> Good product I really liked it<b>(Positive)</b></td><td>Good product really liked <b>(Positive)</b></td></tr>
<tr><td>I didn’t like the product <b>(Negative)</b></td><td>like product <b>(Positive)</b></td></tr>
<tr><td>The product is not good <b>(Negative)</b></td><td>product good <b>(Positive)</b></td></tr>
</table>

We can see the after stopwords the negative reviews also changed to positive.

A bit scary right?

If you are working with basic NLP techniques like BOW, W2V or TF-IDF(Term Frequency and Inverse Document Frequency) then removing stopwords is a good idea because stopwords act like noise for these methods but creating a new list or importing <b>NLP from nlppreprocess</b> is good

```python
from nlppreprocess import NLP
import pandas as pd

nlp = NLP()
df = pd.read_csv('some_file.csv')
df['text'] = df['text'].apply(nlp.process)
```
<center>or</center>
```python
def decontracted(phrase):
    phrase = re.sub(r"\'t", "not", phrase)
    phrase = re.sub(r"\'re", " are", phrase)
    phrase = re.sub(r"\'s", " is", phrase)
    phrase = re.sub(r"\'d", " would", phrase)
    phrase = re.sub(r"\'ll", " will", phrase)
    phrase = re.sub(r"\'ve", " have", phrase)
    phrase = re.sub(r"\'m", " am", phrase)
    return phrase
```
Now, it seems reasonable to use this package for the removal of stopwords and other preprocessing.
Let me know what is your opinion on this in the comment section.

Hope it's useful
A ❤️ would be Awesome 😊