DATASET Analysis and processing
Tags: python, machinelearning, ai, datascience
Published: 2023-05-08 10:30:08

# SEE THIS REPOSITORY AT : [https://github.com/insidbyte/Analysis_and_processing](url)

---


1)-Install python with a version >= 3.9.*

2)-Installing virtual env via pip:

```
pip install virtualenv
```
3)-Access the folder where you want to create the virtual environment and type in command:

```
virtualenv --python C:\Path\To\Python\python.exe name_of_new_venv_folder
```
4)-Access the created folder with the command:

```
cd name_of_new_venv_folder
```
5)-Activate the virtual environment with the command:

```
.\Sripts\activate
```
6)-The following command creates a file called requirements.txt which enumerates the installed packages:

```
pip freeze > requirements.txt
```
7)-This file can then be used by contributors to update virtual environments using the following command:

```
pip install -r requirements.txt
```
8)-To return to normal system settings, use the command:

```
deactivate
```

---

## Analysis_and_processing

**Performs a statistical analysis of the dataset and has more options:**
**1)-Merges two datasets.**
**2)-Performs a first cleanup of the dataset.**
**3)-Analyze and possibly eliminate stop words.**
**4)-Lemmatize.**
**5)-Correct.**

_All the options present in this tool, with the exception of number 3, use multiprocessing._

_As many processes as there are cores of the machine used will start, so it is advisable to run the script from the terminal and to close any activity running on the machine.
The steps must be performed in order otherwise the output dataset will not be reliable!_

## First step:

_We will remove special characters, websites, emails, html code and all the contractures of the English language. First we go to the first file and write on the first line : True._


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bzvju59kyzu1saovijob.png)

_Next we launch main.py._


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pnnq6s5fm8hdes59lqck.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/diwcp3ywuqv4esmd81wn.png)

_Then we insert the following Input:_

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q6cy3j86c0439hi6nnsm.png)

_Output:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/70ipmvzb9y2rtmk22nke.png)

---
## Second step:

_This option corrects the supplied text, dividing it into 8 datasets and concatenating them to return the requested dataset. With 8 cores it took 9 hours for 60MB of datset!! It is highly expensive in terms of: cpu, memory and execution time. I recommend doing this only if necessary._

_Input:_

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0tm1mi7hn3kkotk143eh.png)

_Output:_

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/67onvw665f05fnwsxls8.png)

---

## Third step:

_Let's lemmatize by trimming the dataset a bit and replacing each compound word with its own root_

_Input:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/chpn2ubxyb127fwkevcv.png)

_Output_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7wedwuqg1jr08vuse3q8.png)

---

## Fourth step:

_In this case it is not necessary but in case we have cleaned and lemmatized only the positive or negative reviews, we need to merge the dataset to proceed to the analysis phase._


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tpry5uut4xenos6ptaxj.png)

_As we can see the union dataset weighs less because it has eliminated the stop words separately for positive and negative_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g4666vrugdxpv67w1muc.png)

_After several tests we noticed that the union dataset is less efficient for model generation._

---

## Fifth step:

_This is the most important step because it allows you to greatly lighten the lemmatized and clean dataset. To add new stopwords beyond those already present in the repository just add the words in the text files:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8v09ek0mtwu406v236lh.png)

_Input:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/089ce2z4bhhe8ruitxjh.png)

_Output:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8w2gz3w0ofj6sm355w1p.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jh6g8ia6lve3xsxl3flo.png)

_We can see how many positive and negative reviews the dataset has and perform word-cloud or ngrams analysis. Below are some images that show the effectiveness of the previous phases and some information invaluable for building personalized wordlists._

_Positive and negative review count:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9pj1yquvqttai6u36gw1.png)

_Most Meaningful Words for Word Cloud:_

_Negative:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dzkigljao8qdknfwj3tw.png)

_Positive:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/29su4k0hh3guem5lxcgt.png)

_Most common words in the dataset:_

_Positive:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r5od3m4oley3kvo7p15o.png)

_Negative:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rcdvtbeyrzteqoqxjv8i.png)

_Most common words in the dataset with NGRAMS 2:_


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6lbdf7pfz0rqx37rdabs.png)

---

# CONCLUSIONS:

_At this point we can say that we have created a complete tool that allows us to analyze and modify the dataset. I will show in another repository another useful tool for vectorization and search by tuning hyperparameters with GridSearchCV._

**Link:** [https://github.com/insidbyte/Analysis_And_Generation_Model_ML](url)

**Link documentation of this repository**: [https://scikit-learn.org](url)