Interactive Learning Approach for A rabic Target-Based Sentiment Analysis (2021)

Abstract: Recently, the majority of sentiment analysis researchers focus on target-based sentiment analysis because it delivers in-depth analysis with more accurate results as compared to traditional sentiment analysis. In this paper, we propose an interactive learning approach to tackle a target-based sentiment analysis task for the Arabic language. The proposed IA-LSTM model uses an interactive attention-based mechanism to force the model to focus on different parts (targets) of a sentence. We investigate the ability to use targets, right, and left context, and model them separately to learn their own representations via interactive modeling. We evaluated our model on two different datasets: Arabic hotel review and Arabic book review datasets. The results demonstrate the effectiveness of using this interactive modeling technique for the Arabic target-based task. The model obtained accuracy values of 83.10 compared to SOTA models such as AB-LSTM-PC which obtained 82.60 for the same dataset.

## introduction sentiment analysis (sa) is one of the most prolific research areas in computer sciences, which aims to identify and extract user opinions from reviews. this technique has become an essential part of a wide range of applications in the areas of politics, business, advertising and marketing as it can help in identifying people's opinions towards related targets @xcite . arabic is considered among the top 4 languages in terms of internet usage @xcite . with the rapid growth of arabic web content and low resources for analyzing arabic opinion mining, the need for accurate arabic sentiment analysis tools is very necessary. there are three main classification levels in sentiment analysis: document-level, sentence-level, and target-level. document-level sa aims to classify the sentiment expressed in the whole document. it considers the whole document as a basic information unit (talking about one topic). sentence-level sa aims to classify the sentiment expressed in each sentence. in traditional sentiment analysis, the detailed opinions of all targets of the entity (which are required in many applications) are not provided. to acquire these details, we need to use the target level. target-based sentiment analysis (tbsa) aims to classify the sentiment with respect to the specific targets of entities. the opinion holders can give different opinions for different targets of the same entity, like in this sentence: "the hotel is clean with good services, but the room was too small". target-based sentiment analysis is a finegrained task in sentiment analysis. this kind of fine-grained target-based analysis generally relies on machine learning techniques that require large domain-specific datasets with manual training data @xcite . target-based sentiment analysis has become more popular in recent research as it delivers more accurate results compared to traditional sentiment analysis. given a plain text, the trained model is able to detect the targets that were seen in the training set; the context is simply the sequence of words or tokens around the targets. referring to the previous example, target1: hotel, context: clean, good service, polarity: positive; target2: room, context: too small, polarity: negative. in traditional sentiment analysis (non-target-based sa), this detailed level of analysis is not possible as the analysis is performed at sentence level and hence the entire sentence is classified as either positive or negative. the research area of arabic sa is relatively new. recently, the work on arabic sa has received a lot of attention and a number of papers on traditional sa have been published in the last couple of years @xcite @xcite . however, research work on arabic tbsa has not been addressed in depth yet. most of the existing tbsa research focuses on english @xcite @xcite ) with very little work on other languages such as arabic. in this paper, we address the task of tbsa in arabic. the proposed model uses a neural network with an attention mechanism to force the model to attend to the important parts of a sentence. to achieve that, an interactive attention-based long short-term memory network (ia-lstm) with an interaction technique is used to capture important information related to a given target. previous work in arabic target-based sa used deep learning represented in recurrent neural network (rnn) and lstm models and developed several methods aiming to model contexts through the generation of target-based representations @xcite @xcite . however, the modeling of interactive targets with contexts using attention mechanisms was not addressed. the model proposed in this paper (ia-lstm) is based on a model (ian) proposed for english language tbsa in a previously published paper @xcite that addresses the separately modeled targets and contexts that jointly interact with each other with an attention mechanism. however, the left context was ignored in the previously proposed approach in @xcite . given the fact that arabic language is written from right to left, the addition of the left context may improve the performance as the most of the opinionated words come after the target (left context). the proposed model (ia-lstm) is obtained by adding the left context to the ian model from @xcite , to end up with a model that considers the three main elements (left context, target, and right context). code for the proposed model is publicly available on github 1 . the rest of this paper is organized as follows: section 2 presents the related work of sa and tbsa in english and arabic. section 3 discusses the methodology used for the proposed model in this paper. section 4 describes the datasets used for evaluations. section 5 explains the baselines, evaluations, the experimental results and analysis. section 6 concludes this paper. ## related work the research work on arabic tbsa has few examples as the published papers are limited, so, the 1 https://github.com/hustud/ia-lstm first subsection of related work mainly focus on traditional arabic sa while the second subsection covers the published work on tbsa on both arabic and other languages. ## proposed approach recurrent neural networks (rnns) are deep learning neural networks designed specifically to learn sequences of data and are mainly used for textual data classification. the learning process is done at hidden recurrent nodes depending on their previous layers of nodes. however, rnns suffer from the vanishing gradient problem when handling long sequences of data. bi-directional long short-term memory (bi-lstm) @xcite was proposed as a solution for this problem and have proven to be efficient in many nlp-related problems. in contrast to the standard rnns, bi-lstm units have a major role in extracting and learning important features out of the input or computed data and keeping the computed values as long as they are needed in the memory vector. in this research, we focus on target-based sentiment polarity classification considering the right and left contexts. the enhancement of sentiment classification performance can be achieved by considering the targets and their contexts. the good performance relies on simultaneously modeling targets and contexts precisely. targets and contexts can influence the representation of each other. for example, the target word "hotel" can be naturally associated with the context word "clean" and vice versa. therefore, targets and contexts can be modeled individually but learn from their interactions. in the input text, each word usually has its own contribution or importance, which is different from other words in the final representation for sentiment analysis. for instance, the importance of the word "room" is higher in the representation of the target "room price", which is described by "expensive". therefore, in the proposed model, the attention weights for both targets and contexts are computed to respectively capture their important information. the proposed model is an interactive attention lstm-based (ia-lstm) model which employs long short-term memory networks (lstm) and attention mechanisms. to get important information from the left and right context, the model uses an attention mechanism associated with a target then computes context representation for sentiment polarity identification. in addition, the proposed model makes use of the interactive information from the word's context to supervise the target modeling. finally, the model concatenates the attended left context representations, target representations, and right context representations, then uses them to predict the sentiment polarity (see fig. 1 ). following the model notation in fig. 1 , assume that a left context consists of k words [w lc1 , w lc2 , ..., w lck ], a target consists of m words [w t1 , w t2 , ..., w tm ], and a right context consists of n words [w rc1 , w rc2 , ..., w rcn ]. we use pre-trained word embeddings for word representation of contexts and targets. then, since there is a strong dependence between words in a sentence, the lstm network is used to learn the hidden word semantics as lstm is good at learning long-term dependencies. next, lstm produces the hidden states [h lc1 , h lc2 , ..., h lck ] for the left context words, [h t1 , h t2 , ..., h tm ] for the target words, and [h rc1 , h rc2 , ..., h rcn ] for the right context as word representations. then, the model calculates the average of the hidden states to get the initial representations of the contexts and target. the attention mechanism in the proposed model is adopted by using the initial representations of the contexts and target as input to help in selecting important information for classifying the sentiment polarity. to form the attention mechanism, we use the average as well as the last hidden state output of target and contexts to capture an abstract representations of the input sequence. the use of the average value is to form the initial representation of the other component (target or context) and combine it with the last hidden state of the current component. we found this is the best way to form the attention mechanism and it help to reduce the noise and the sparse information associated with the arabic language @xcite . the attention process is described by considering the left context, target, and the right context, as shown in fig. 1 . using the left context words representation [h lc1 , h lc2 , ..., h lck ] and the average of hidden states of target representation t avg , the model computes the left attention vector lα i . similarly, the model computes the target attention vector β i by using the average of left context words lc avg , the target words representation [h t1 , h t2 , ..., h tm ], and the average of the right context vector rc avg . the same technique is followed for the right attention vector rα i obtained using the right context words representation [h rc1 , h rc2 , ..., h rcn ] and the average of hidden states of target representation t avg . then, the attention vectors lα i ,β i , and rα i are used to obtain the word attention weights and concatenated as one vector before being fed to the softmax classifier. formally, given the word embeddings of the contexts and target, the hidden state representations of the contexts (left and right) and target are [h lc1 , h lc2 , ..., h lck ], [h rc1 , h rc2 , ..., h rcn ] and [h t1 , h t2 , ..., h tm ] respectively. the initial representation of contexts (left -equation (1) and right -equation ( 2 ) ) and targets equation ( 3 ) is obtained by calculating the average of each as follows: to make the model focus on the important parts of the representations, an attention mechanism that employs the initial representations of the contexts and target is used. the target influence on the left context and the influence of the left context on the target is considered, as well as the influence of the target on the right context and the influence of the right context on the target. the left attention vector lα i is generated using both the left context and target representations as in equation ( 4 ): where γ is a score function that computes the importance of h lc i and h rc i in the left and right context respectively. this function is defined as equation ( 5 ): where w is a weight matrix, b is a bias, and t avg t is a transpose of t avg . the right attention vector rα i is generated using both the right context and target representations as equation ( 6 ): similarly, the target attention vector β i is calculated using all of the right context, left context, and the target as (7): the final representation is done by concatenating the representations of the three components of the left context, target, and the right context (c l , t , and c r respectively) as one vector (f). finally, the prediction of sentiment polarity (positive, neutral, negative) is done by using the softmax layer as in equation ( 11 ): where w l is the weight matrix and b l is the bias. ## datasets there are two benchmark datasets were used to evaluate the proposed approach, arabic hotel reviews and arabic book reviews datasets. ## evaluation we evaluated the proposed model on the arabic hotel review and the arabic book review. the evaluation is done by computing the accuracy of the polarity. the used accuracy metric defined as the number of correctly predicted polarity labels of the (gold) targets, divided by the total number of the gold targets. the test set contains sentences without polarity labels which is expected to be predicted by the model and compared with the labels in the test gold set containing the same sentences. we compared the proposed model performance with the related work and sota approaches stated as lstm @xcite , ian @xcite , insight-1 @xcite , and ab-lstm-pc (al-smadi et al., 2019) evaluated on the same arabic hotel review dataset. for the arabic book review dataset, the proposed model is compared with a subset of these approaches: lstm @xcite and ian @xcite . in addition, the baselines labelled as "baseline" in ## conclusion in this paper, we proposed a deep learning-based approach to tackle arabic tbsa. the proposed model "ia-lstm" uses an interactive attentionbased technique for the task. the main idea of the proposed ia-lstm model is to use three attention networks to interactively model the target and contexts (left and right). it is based on previous work in english that uses only the right context but given the way arabic is written from right to left, the addition of the left context provides important context information. the model can focus on the important parts in the sentence and identify the sentiment polarity. the proposed approach was evaluated on two different datasets: arabic hotel reviews and book reviews. experiments verify that the proposed approach outperforms the baselines and other approaches evaluated on the same datasets. implementing the interactive attention-based model demonstrated that the model can learn effective features for targets and contexts (left and right) and enhance the performance of arabic tbsa.