Sentiment Analysis
Tags: machinelearning, nlp, datascience
Published: 2023-03-25 12:23:26

## Getting Started With Sentiment Analysis

It is the process of detecting positive or negative sentiment in text.
It is also referred to as opinion mining.
It is an approach to natural language processing (NLP) that identifies the emotional tone <br>
behind a body of text.
It is vastly used by organizations to determine and categorize opinions about a produt, service or idea<br>

Sentiment analysis involves the use of <em>data mining, machine learning (ML), artificial intelligence<br>
and computational linguistics</em> to mine text for sentiment and subjective information.<br>
Such information maybe classified as:
- positive
- neutral
- negative
This classification is also known as ***polarity*** of a text.
***Graded Sentiment Analysis***
- very positive
- positive
- Neutral
- Negative
- Very Negative
This is also referred to as graded or fine-grained sentiment anlysis.

Types of Sentiment Analysis
- Intent-based - recognizes motivation behind a text
- Fine-grained - graded sentiment analysis
- Emotion-detection - allows detection of various emotions
- Aspect-based - anayses text to know particular aspects/features mentioned in all the polarity.

We will not dive into these types for now.

This in turn helps organizations to gather insights into real-time customer sentiment,<br> 
customer experience and brand reputation.<br>
Generally these tools use text analytics to analyze online sources .

***Benefits of sentiment analysis***
- sorting data as scale
- real-time analysis
- consistent criteria

## Steps involved in Sentiment Analysis

Sentiment analysis generally follows the following steps:
- ***Collect data*** - The text to be analyzed is identified and collected.
- ***Clean the data*** - The data is processed and cleaned to remove noise and parts of speech<br> 
that don't have meaning relevant to the sentiment of the text.
- ***Extract features*** - A machine learning algorithm automatically extracts text features <br>
to identify negative or positive sentiment.
- ***Pick an ML model*** - A sentiment analysis tool scores the text using rule-based,<br>
automatic or hybrid ML model.
- ***Sentiment classification*** - Once a model is picked an used to analyze a piece of text,<br>
it assigns a sentiment score to the text including positive, negative of neutral.


Let's have a  deep dive in sentiment analysis using an example

## Step 1. Collect Data
We are going to used a data set from UCI Machine Learning Repository.

Let's start with importing the libraries that we will be using:
`punkt` is a data package that contains pre-trained models for tokenization.

    # import the required packages and libraries
    import numpy as np
    import pandas as pd
    import nltk
    nltk.download('punkt')

### loading the dataset

    pd.set_option('display.max_colwith', None)
    df = pd.read_csv('https://gist.githubusercontent.com/fmnobar/88703ec6a1f37b3eabf126ad38c392b8/raw/76b84540ccd4b0b207a6978eb7e9d938275886ff/imdb_labelled.csv')
    df.head()

**Output**

![header for data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vfe0j0zsyebssaqkofev.png)

We can now see that there are only two columns `text` and `label`.
The `label` indicates the sentiment of the review
- 1 indicates a postive sentiment
- 0 indicates a negative sentiment.
This thus indicates the polarity of the sentiment.

We now create a sample string, which is the first entry in the `text` column of the dataframe `df`.

    sample = df.text[0]
    sample

**Output**

![sample](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t9aj11abhp4fxi1jqhvm.png)

## Tokens and Bigrams
### a. Tokens
A `token` is a single unit of meaning that can be identified in a text.
It is also known as a `unigram`.
`Tokenization` is the process of breaking down a text into individual tokens.
The functions that perform `tokenization` are called `tokenizers`.
This concept is implemented with the `nltk.word_tokenize` function.
- the function takes a string of text as input and returns a list of tokens.
- it splits the text into individual words and punctuation marks.
Let's see an example the functions usage by tokenizing the `sample` text.

    sample_tokens = nltk.tokenize(sample)
    sample_tokens[:10] # view a list of elements upto the 10th token

**Output**

![Sample Tokens](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z6gbggsopnnwbgg7rfjv.png)

### b. Bigrams
If we combine two unigrams/tokens we form a `bigram`.
A bigram is a pair of adjecent tokens in a text.
They are used to capture some of the context in which a particular word <br/> or phrase appers.
They are used to build statistical models of language which are <br> sequences of n words/tokens.
By analyzing the frequency of different n-grams in a large corpus of text,<br/>NLP systems can learn to predict the probability of dofferen words occuring in a particular context.

`bigrams` are implememted with the `nltk.bigrams` function

Let's see this in action

    sample_bitokes = list(nltk.bigrams(sample_tokens))
    
    # Return the first 10 bigrams
    sample_bitokens[:10]

**Output**
![bitokens](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/np3cytthignf61dxaa6d.png)

## Frequency Distribution
Refers to the count or proportion of words or prases asscociated with positive or negative sentiment.
It basically counts the occurrence of each sentiment-bearing word/phrase<br/> and then calculate the frequency distribution.

implemented using the `nltk.FreqDist` function

***What are the top 10 most frequently used tokens in our sample?***

    sample_freqdist = nltk.FreqDist(sample_tokens)
    
    # Return the top 10 most frequent tokens
    sample_freqdist.most_common(10)

**Output**

![freqDist](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ezwyje8yo44qnsvi0u5v.png)

This results ultimately make sense:
- a comma, `the` , `a` or periods can be quite common in a phrase.

Let's create a function named `tokens_top` that takes in a text <br/> as input and returns the top n most common tokens in a given text.

    def tokens_top(text, n):
        # create tokens
        tokens = nltk.word_tokenize(text)
        
        # create the frequency distribution
        freqdist = nltk.FreqDist(tokens)
        
        # return the top n most common tokens
        return freqdist.most_common(n)
    
    # Call the function 
    tokens_top(df.text[1], 10)

**Output**

![def freqdist](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/myokkh3r9yvekmk8uuq3.png)

### Document-Term Matrix
It is a matrix that represents the frequency of terms that occur in a collection of documents.
The rows represent the documents in the corpus and the columns represent the terms .
The cells of the matrix represents the frequency or weight of each term.

We can implement this with `scikit-learn's` `CountVectorizer`

**Example**

    #import the package
    from sklearn.feature_extraction.text import CountVectorizer
    
    def create_dtm(series):
        # Create an instance/object of the class
        cv = CountVectorizer()
        
        # create a dtm from the series parameter
        dtm = cv.fit_transform(series)
        
        # convert the sparse array to a dense array
        dtm = dtm.todense()
        
        # get column names
        features = cv.get_feature_names_out()
        
        # create a dataframe
        dtm_df = pd.DataFrame(dtm, columns = features)
        
        # return the dataframe
        return dtm_df
    # Call the function for df['text].head
    create_dtm(df['text'].head())

**Output**

![dtm](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ws4xtm0vlbuxjrx1z5s.png)

## Data Cleaning
### Feature Importance
Refers to the extent to which a specific feature/variable contributes to the <br/>prediction or classification in sentiment analysis.

There are differet methods that can be used to determine feature importance:
- machine learning algorithms eg. decision trees and random forests
- statistical methods eg. correlation or regression analysis

feature importance is a useful tool in sentiment analysis as it can help identify <br/>the most important features for accurately predicting the sentiment of a text.


**Example**
we'll define a function "top_n_tokens" that has 3 parameters
`text`, `sentiment` and `n`<br>
the function will return the top `n` most important tokens
to predict the sentiment of the text.<br>
We'll use `LogisticRegression` from `sklearn.linear_model`
with the following parameters:
- `solver = 'lbfgs'`
- `max_iter = 2500`
- `random_state = 1234`

    from sklearn.linear_model import LogisticRegression
    
    def top_n_tokens(text, sentiment, n):
        # create an instance of the class
        lgr = LogisticRegression(solver = 'lbfgs', max_iter = 2500, random_state = 1234)
        cv = CountVectorizer()
        
        # create the DTM
        dtm = cv.fit_transform(text)
        
        # fit the logistic regression model
        lgr.fit(dtm, sentiment)
        
        # get the coefficients
        coefs = lgr.coef_[0];
        
        # create the features/column names
        features = cv.get_features_names_out()
        
        # create the dataframe
        df = pd.DataFrame({'Tokens' : features, 'Coefficients' : coefs}) 
        # return the largest n
        return df.nlargest(n, coefficients)
        # Test if on df['text]
        top_n_tokens(df.text, df.label, 10)

**Output**

![feat importance](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4kml7yu934vr5uf6d26s.png)

To validate the hypothesis that the most important features will be the ones that<br> indicate a strong positive sentiment, let's look at the 10 smallest coefficients.

    from sklearn.linear_model import LosticRegression

    def bottom_n_tokens(text, sentiment, n):
        # create an instance of the class
        lgr = LogisticRegression(solver = 'lbfgs', max_iter = 2500, random_state = 1234)
        cv = CountVectorizer()
        
        # create the DTM
        dtm = cv.fit_transform(text)
        
        # fit the logistic regression model
        lgr.fit(dtm, sentiment)
        
        # get the coefficients
        coefs = lgr.coef_[0];
        
        # create the features/column names
        features = cv.get_features_names_out()
        
        # create the dataframe
        df = pd.DataFrame({'Tokens' : features, 'Coefficients' : coefs})
        
        # return the smallest n
        return df.nmallest(n, coefficients)
    # Test if on df['text]
    bottom_n_tokens(df.text, df.label, 10)

**Output**

![feat importance](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ctsfoui3h7cqdik1w60g.png)

In the example that we've covered till this far we've used labelled data<br/> What if we do not have labelled data?
Then we can use pre-trained models such as:
- TextBlob
-VADER
- Stanford ColeNLP
- Google Cloud Natural Language API
- Hugging Face Transformers

Let's explore `TextBlob`

## TextBlob
It is a Python library that provides a simple API for performing common <br/>NLP tasks such as sentiment analysis.
It uses a pre-trained model to assign a sentiment score to a piece of text, ranging from -1 to 1

It is built on top of NLTK (natural language toolkit)
It also provides additional information such as:
- subjectivity score

It returns the sentiment of agiveen data in the format of a named tuple as follows:
    `(polarity, subjectivity)`

polarity score is a float within the range of [-1.0, 1.0].
- it aims at differentiating whether the text is positive or negative

subjectivity is a float within the range [0.0, 1.0]
- 0.0 is very objective
- 1.0 is very subjective

TextBlob also provides other features such as:
- part-of-speech tagging 
- a noun phrase extraction

### Example

Let's define a function named `polarity_subjectivity` that accepts two argument.
The function uses `TextBlob` to the provided `text`
if `print_results` = True, prints polarity and subjectivity of the text elseM<br/> returns a tuple of float values 1st being polarity and 2nd being subjectivity

You can install `TextBlob` using

    !pip install textblob

    #import TextBlob
    from textblob import TextBlob
    
    def polarity_subjectivity(text = sample, print_results = False):
        # create an instance of TextBlob
        tb= TextBlob(text)
        
        # if condition is metm print the results
        if print_results:
            print(f"Polarity is {round(tb.sentiment[0], 2)} : Subjectivity {round(tb.sentiment[1], 2)}")
        else:
            return (tb.sentiment[0], tb.sentiment[1])
    
    # Test the function
    polarity_subjectivity(sample, print_results =  True)

**Output**

![pol_sub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jzgzu56twiv8c11v0dk0.png)

The results indicate that our sample has a slight positive polarity and it's relatively subjective thought not by  a high degree

Let's define a function `token_count` that accepts a string and using  `nltk's word_tokenizer`,<br/>returns an integer number of tokens in the given string

Then define another function `series_tokens` that accepts a Pandas Series as argument <br/> and aplies the function
`token_count` to the given series.
Use the second function on the top 10 rows of our dataframe

    # import libraries
    from nltk import word_tokenize

    # Define the first function that counts the number of tokens in a given string
    def token_count(string):
        return (len(word_tokenize(string)))

    # Define the second function that applies the           token_count funnction to a given Pandas series
    def series_tokens(series):
        return series.apply(token_count)

    # Apply the function to the top 10 rows of the data frame
    series_tokens(df.text.head(10))

**Output**

![pol series](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/26ij4mkzwpe99mr0wjjj.png)

Let's define a function named `series_polarity_subjectivity`
that applies the `polarity_subjectivity` function we defined earlier

    # define the function
    def series_polarity_subjectivity(series):
        return series.apply(polarity_subjectivity)
    
    # apply to the top 10 rows of df['text']
    series_polarity_subjectivity(df['text'].head(10))

**Output**


## Measure of Complexity - Lexical Diversity

Lexical diversity refers to the variety of words used in a piece of writing or speech.
It is a measure of how often different words are used in a given text or speech and is often used as an indicator  of the richnes and complexity of vocabulary.
It thus defines the number of unique tokens over the total number of tokens.

### Example
Let's define a `complexity` function that accepts a string as an argument and returns the lexical complexity score defined as the number of unique tokens over the total number of tokens.

    def complexity(string):
        # create a list of all tokens
        total_tokens = nltk.word_tokenize(string)
        
        # create a set of words(It keeps only unique values)
        unique_tokens = set(total_tokens)
        
        # Return the complexity measure
        if len(total_tokens) > 0:
            return len(unique_tokens) / len(total_tokens)
    
    # apply the function to top 10 rows
    df.text.head(10).apply(complexity)

**Output**
![lexical Diversity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9gzaa9ymryix9572ry9j.png)

Some interesting insights the row at index 3 and 4 have the highest lexical diversity. All the tokens in them are totally unique.

## Text Cleanup - Stopwords and Non-alphabeticals
This step ensures that the text data is in a constitent format and to remove noise,<br/>
irrelevant information and other inconsitencies.
Some of the techniques for text cleanup:
- Lowercasing
- Tokenization
- Stopword Removal
- Removing Punctuation
- Stemming and Lemmatization
- Removing URL's and mentions
- Removing emojis and emotions

### Example

    #import the library
    from nltk.corpus imort stopwords
    
    # Select only English stopwords
    english_stop_words = stopwords.words('english')
    
    # print the first 20
    print(english_stop_words[:20])

Let's look at an example to remove non-alphabetical
We'll use `isalpha`

    string_1 = "Crite_Jes.cd"
    string_2 = "a quick dog"
    string_2 = "We are good!"
    
    print(f"String_1: {string_1.isalpha()}\n")
    print(f"String_1: {string_2.isalpha()}\n")
    print(f"String_1: {string_3.isalpha()}\n")

**Output**


![Clean aplhabets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gsppy2pfej8lly71rt2g.png)