Text Mining and Movie Review Classification through RapidMiner.
Tags: machinelearning, textmining, rapidminer
Published: 2019-10-11 01:52:38

One of the most complex tasks in supervised classification methods is to have a tagged data set that allows you to train a model in the first instance and then validate it. Within the text mining process there are few corpus that can be used for these processes. Therefore, the objective will be to create a set of tagged data that allows to form a classification model to recognize the positive or negative reviews of a set of films.

Data source: [https://www.cinemablend.com/reviews/](https://www.cinemablend.com/reviews/)

Project repositories: [https://github.com/esdanielgomez/MovieClassificationRMP/](https://github.com/esdanielgomez/MovieClassificationRMP/)

#Part 1: Collection of unstructured data.#

For this process we begin with the extraction of the links of the comments of the films (1470 in total) through 30 pages on the site.

***REPOSITORY: Final_A***

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/8vwe7jyidrhjisvtauip.jpeg)

In this case, both the comment link and its star rating are being selected (weighted between 1.0 and 5.0):

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/lub5dqpjiv9b6z607g53.jpeg)

***REPOSITORY: Final_B***

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/cx8l5jozv1ssqpxklvpy.jpeg)

Now as a next step, the comments are divided into two sections: Positive and Negative:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/kolbkvz71j2zxjrtuj5q.jpeg)

For negative comments, the selection of comments with a rating of less than or equal to 3.0 is made:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/q4nsv7gc3cdmq86ue1mj.jpeg)

And for negative comments with a rating greater than 3.0:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/8cl5klpd285h5yr30nws.jpeg)

Subsequently, the information of the films is obtained, in which the most relevant for this study is the Comment attribute:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/14huwmr1xuffvbstlwrv.jpeg)

The process for obtaining comments is as follows:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/e0pgm8pwy17g1esup6hl.jpeg)

Finally, these comments are stored in two Excel files (datasetComentariosNegativos.xlsx and datasetComentariosPositivos.xlsx).

***REPOSITORY: Final_C***

Next, 650 movie comments are selected per dataset (Positive and Negative) and two new datasets are prepared for training data and data for subsequent tests.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/2c20ianm8pnlv8toc1gc.jpeg)

#Part 2: Preparation of the data.#

Until now the data is organized in this way:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/4xwgxobyj8b2r9o7dh9y.jpeg)

***REPOSITORY: Final_D***

Next, pre-processing activities are executed to eliminate HTML tags, empty words in English and through a custom dictionary, among other processes:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/po3alzz7av3dx8yo6tcc.jpeg)

Process Documents from Data:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/d5yj62m8ybm171knxz58.jpeg)

#Part 3: Identify key characteristics.#

***REPOSITORY: Final_D (In the same repository as part 2).***

The result of the previous process (Part 2) returns a structured table consisting of one row for each entry extracted from the website and a number of columns indicating the different tokens that are part of the document and their occurrence weight. However, trying to execute a learning process with so many variables is a very expensive task. Therefore, in this process the objective is to filter those columns (tokens) by some method of selecting characteristics.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/m0ybknsi8g7gu7q4tozr.jpeg)

For this, the Weight of Information Gain operator is used to classify the characteristics by their relevance to the tag attribute based on the information gain ratio.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/4m278cag311plw8nju84.jpeg)

Also, the Select by Weights operator is used which selects only those attributes of an input set whose weights meet the specified criteria with respect to the input weights. In this case, through the top k attribute in which the best 90 characteristics of the data set are selected.

#Part 4 and 5: Build the training model and apply it.#

Two algorithms are used to build the training model: Naive Bayes and SVM, in which you can identify which provides the best accuracy and performance.

Also, within the same process, the two trained models are applied to the test data and their accuracy is verified (The data in this test dataset is pre-processed in the previous parts).

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/t0q0867w7yln1u7jpvra.jpeg)

Model and validation NB (Naive Bayes)

![](https://thepracticaldev.s3.amazonaws.com/i/7tajohw7912rhvj5p6bx.jpeg)

Model and validation SVM (Support Vector Machine)

![](https://thepracticaldev.s3.amazonaws.com/i/dq181ta9yqi2058c6r8a.jpeg)