CountVectorizer vs TFIDF - Logistic Regression
Tags: nlp, machinelearning, ai, python
Published: 2023-03-28 10:35:37

Recently I have become curious about how Natural Language Processing(NLP) works. If you are someone like me then this blog could be really helpful.

When beginning with M.L. we would have observed how tabular data was used to train an M.L. model, most of the columns would be numeric columns, rest of the text columns would usually be having 1 word which would be converted to numbers using techniques like one hot encoding.

There are cases where columns have sentences or even paragraphs so new techniques should be applied to convert raw text data to computer-usable form and in this blog we are going to see 2 such ways to do it.

## What is Vectorization and why Vectorization
Many machine learning algorithms and almost all deep learning algorithms are not capable of processing text in the raw form, they need numerical inputs. This process of converting text data to numerical data is called **vectorization**. In the NLP world this process is referred to as **embeddings**.

## CountVectorizer
When we use countvectorizer we create a **sparse matrix**, and in this sparse matrix we store **count** of all the **words** in our [corpus](https://dictionary.cambridge.org/dictionary/english/corpus), this is a simple but efficient way of converting text to numerical data.
Each row in the sparse matrix contains the **word** and its corresponding **count** in that particular line.


![Count Vectorizer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sq0s0hg986azrkvzcy1m.png)
In the above diagram, you could observe that in the **x-axis** all **distinct words** are present and in the **y-axis** the **sentence index**/**line index** is present(which is represented as doc), this is the sparse matrix representation and this is used to train the M.L model.

Fortunately [scikit-learn](https://scikit-learn.org/stable/) has an inbuilt module that we could use to generate the sparse matrix and is super easy to use.
 
```python
from sklearn.feature_extraction.text import CountVectorizer
from nltk.tokenize import word_tokenize
```
In addition to importing the **CountVectorizer** module, we also importing **word_tokenize**, this is nothing but an inbuilt module which could be used to convert each sentence into chunks of words and this is a preprocessing step in almost all NLP algorithms.

Let's create our own sample input
```python
corpus = [
    "hello, how are you, I am Praveen?",
    "You know football is a wonderful sport, what do you think?",
    "Opensource is something that everyone should appreciate, what do you think?"
]
```
This is our sample input

```python
ctv = CountVectorizer(tokenizer = word_tokenize, token_pattern=None)
```
We are creating an object and the argument **tokenizer** is assigned with **word_tokenize** module so all words and all special characters would be considered as separate tokens.

```python
ctv.fit(corpus)
corpus_transformed = ctv.transform(corpus)
``` 
The **corpus_transformed** variable now holds the sparse matrix, now we can visualize the sparse matrix.

![Count Vectorizer Sparse Matrix](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hqq343sle1tujpwg3kyo.png)
Our input contains 3 lines so 0,1 and 2, and the other number in the tuple-like structure is the unique id of each word in the corpus.
```python
print("Unique index assigned to each word : ",ctv.vocabulary_)
```  

![Unique index - CV](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jxs7ate6qvvdtpjseri0.png)
**Unique index assigned to each word :  {'hello': 8, ',': 0, 'how': 9, 'are': 4, 'you': 20, '?': 1, 'know': 11, 'football': 7, 'is': 10, 'a': 2, 'wonderful': 19, 'sport': 15, 'what': 18, 'do': 5, 'think': 17, 'opensource': 12, 'something': 14, 'that': 16, 'everyone': 6, 'should': 13, 'appreciate': 3}
**

## Term Frequency Inverse Document Frequence (TFIDF)

![TFIDF](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0xsy94rl4gst3cs27342.png)
The above picture is self explanatory for the TFIDF formula, this method too creates a sparse matrix but instead of the count here we have the **TFIDF formula applied to each token**, the resultant is a **float** value.

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.tokenize import word_tokenize
```
Importing the necessary modules

```python
corpus = [
    "hello, how are you?",
    "You know football is a wonderful sport, what do you think?",
    "Opensource is something that everyone should appreciate, what do you think?"
]
```
Input data that we are going to use


```python
# To also include special characters while creating sparse matrix
tfidf = TfidfVectorizer(tokenizer = word_tokenize, token_pattern=None)
tfidf.fit(corpus)
```
Creating an object and fitting it on our input data.
```python
# TFIDF vectorizer
corpus_transformed = tfidf.transform(corpus)
```
**corpus_transformed** contains the sparse matrix generated which could be used to train the model.

```python
print("Sparse Matrix Representation : ", corpus_transformed)
```

![TFIDF matrix](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/flheo8d36brx67mwau68.png)

```python
print("Unique index assigned to each word : ", tfidf.vocabulary_)
```
*
Unique index assigned to each word :  {'hello': 8, ',': 0, 'how': 9, 'are': 4, 'you': 20, '?': 1, 'know': 11, 'football': 7, 'is': 10, 'a': 2, 'wonderful': 19, 'sport': 15, 'what': 18, 'do': 5, 'think': 17, 'opensource': 12, 'something': 14, 'that': 16, 'everyone': 6, 'should': 13, 'appreciate': 3}

*

## Lets use the sparse matrix generated in Logistic Regression
Let's use a kaggle dataset to perform logistic regression, the dataset that we are going to use is [IMDB movie review](https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews) to perform sentiment classification - positive/negative

```python
import pandas as pd
from nltk.tokenize import word_tokenize
from sklearn import linear_model
from sklearn import metrics
from sklearn import model_selection
from sklearn.feature_extraction.text import CountVectorizer
```
Importing the necessary modules

```python
if __name__ == "__main__":
    df = pd.read_csv("/home/praveen/Desktop/Projects/Approching_Almost_Any_ML_Prob_Book/NLP/data/IMDB Dataset.csv")

    # Converting sentiment to 1 and 0
    df.sentiment = df.sentiment.apply(lambda x: 1 if x == 'positive' else 0)

    df["kfold"] = -1

    df = df.sample(frac=1).reset_index(drop=True)

    y = df.sentiment.values

    kf = model_selection.StratifiedKFold(n_splits=5)

    for f, (t_, v_) in enumerate(kf.split(X=df, y=y)):
        df.loc[v_, 'kfold'] = f

```
We are converting the **sentiment** column which is our target column to 0 and 1. We are going to take up the [k-fold cross validation](https://www.analyticsvidhya.com/blog/2022/02/k-fold-cross-validation-technique-and-its-essentials/) approach, this is nothing but converting the total dataset into k(for eg 4) folds and during training 3 folds would be used for training and 1 fold would be used for validation, this is a better way to evaluate our model's performance.

There is something called **StratifiedKFold** where all the folds would have **balanced distribution of classes**.
We are adding a new column to the dataframe called kfold to represent which fold the particular data point or row belongs to.

```python
    accuracy_list = []
    for fold_ in range(5):
        train_df = df[df.kfold != fold_].reset_index(drop=True)
        test_df = df[df.kfold == fold_].reset_index(drop=True)

        count_vec = CountVectorizer(tokenizer=word_tokenize, token_pattern=None)
        count_vec.fit(train_df.review)

        xtrain = count_vec.transform(train_df.review)
        xtest = count_vec.transform(test_df.review)

        model = linear_model.LogisticRegression()

        model.fit(xtrain, train_df.sentiment)

        preds = model.predict(xtest)

        accuracy = metrics.accuracy_score(test_df.sentiment, preds)
        accuracy_list.append(accuracy)

        print(f"Fold : {fold_}")
        print(f"Accuracy : {accuracy}")
        print("")

    for i in range(0, 4):
        print(f"Fold : {i+1}, Accuracy : {accuracy_list[i]}")
``` 

We are using countvectorizer to transform the review to vectors and using logistic regression to classify the sentiment. We are using **accuracy** to evaluate our model performance.

![LR accuracy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x5mmyjanrskonp4v6i22.png)  
We could observe that the model is able to reach almost **90 percent accuracy**, that was easy right?

### Same implementation using TFIDF
```python
import pandas as pd
from nltk.tokenize import word_tokenize
from sklearn import linear_model
from sklearn import metrics
from sklearn import model_selection
from sklearn.feature_extraction.text import TfidfVectorizer

if __name__ == "__main__":
    df = pd.read_csv("/home/praveen/Desktop/Projects/Approching_Almost_Any_ML_Prob_Book/NLP/data/IMDB Dataset.csv")

    # Converting sentiment to 1 and 0
    df.sentiment = df.sentiment.apply(lambda x: 1 if x == 'positive' else 0)

    df["kfold"] = -1

    df = df.sample(frac=1).reset_index(drop=True)

    y = df.sentiment.values

    kf = model_selection.StratifiedKFold(n_splits=5)

    for f, (t_, v_) in enumerate(kf.split(X=df, y=y)):
        df.loc[v_, 'kfold'] = f

    
    accuracy_list = []
    for fold_ in range(5):
        train_df = df[df.kfold != fold_].reset_index(drop=True)
        test_df = df[df.kfold == fold_].reset_index(drop=True)

        tfidf_vec = TfidfVectorizer(tokenizer=word_tokenize, token_pattern=None)
        tfidf_vec.fit(train_df.review)

        xtrain = tfidf_vec.transform(train_df.review)
        xtest = tfidf_vec.transform(test_df.review)

        model = linear_model.LogisticRegression()

        model.fit(xtrain, train_df.sentiment)

        preds = model.predict(xtest)

        accuracy = metrics.accuracy_score(test_df.sentiment, preds)
        accuracy_list.append(accuracy)

        print(f"Fold : {fold_}")
        print(f"Accuracy : {accuracy}")
        print("")

    for i in range(0, 4):
        print(f"Fold : {i+1}, Accuracy : {accuracy_list[i]}") 

```

![TFIDF accuracy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3vqa742xyorvoxycknee.png)
TFIDF is also close to **90 percent accuracy**.

With this I conclude, so in this blog we have seen how sentence/word is converted to vector and how it is used to tackle a classification problem.

Github : https://github.com/praveenr2998/Approching_Almost_Any_ML_Prob_Book/tree/main/NLP
Book : https://github.com/abhishekkrthakur/approachingalmost/blob/master/AAAMLP.pdf (a generous author :)))
Linked in : [profile](www.linkedin.com/in/praveenr2998)

See you in the next blog, bye...