Casting the Same Sentiment Classification Problem (2021)

Abstract: We introduce and study a problem variant of sentiment analysis, namely the “same sentiment classification problem”, where, given a pair of texts, the task is to determine if they have the same sentiment, disregarding the actual sentiment polarity. Among other things, our goal is to enable a more topic-agnostic sentiment classification. We study the problem using the Yelp business review dataset, demonstrating how sentiment data needs to be prepared for this task, and then carry out sequence pair classification using the BERT language model. In a series of experiments, we achieve an accuracy above 83% for category subsets across topics, and 89% on average.

## introduction at the sixth argument mining workshop argmining 2019 @xcite , the same side stance classification problem has been introduced by @xcite as a shared task to the argument mining community. identifying the stance of an argument towards a topic is a fundamental problem in computational argumentation. the task presents a new problem variant, namely to classify whether two arguments share the same stance without the need to identify the stance itself. the underlying hypothesis is that this can be achieved in a topic-agnostic manner since only the similarity of two given arguments needs to be assessed. similarly, in the authorship analysis community, the authorship verification problem @xcite is the task of determining for a given pair of texts whether they have been written by the same author. here, too, instead of classifying a given text into predefined author classes, as is the case with authorship attribution, the verification problem casts the problem as a pairwise similarity-based classification task. in this paper, we recast sentiment analysis in the same manner: given two texts of unknown sentiment polarity, determine whether their sentiment is the same. unlike for the same side and the same author classification problems, which suffer from a lack of large-scale training data, due to many resources available for sentiment analysis, scaling up does not prove to be a problem for the same sentiment problem. we see three major contributions in studying this task variant: (1) focused research on topic-agnosticity, enabling direct observations of the effect of topic and that of agnostic modeling. (2) potentially easing generalization across domains. (3) in time, a new paradigm of approaches may emerge (whereas the prevailing one still rules today). our contributions are as follows: we demonstrate how to prepare standard sentiment data for meaningful training and evaluation, introduce an approach based on the transformer neural network architecture where we adapt the sequence pair classification task to the same sentiment problem, and evaluate our model in various experiments. 1 in what follows, section 2 reviews related work, section 3 introduces our approach and explains the dataset and its preparation, and section 4 reports on our evaluation. ## related work sentiment analysis has a wide range of applications in many languages and a variety of methods were developed to refine results and adapt to use-cases @xcite . its main task is to determine the opinion or attitude of an author, either a single person or a group, about something, be it a product, brand, or service @xcite . it has importance for businesses, in campaigns, and the financial sector, among others, and as a result, it has undergone much research to improve accuracy using different models and forms of data representation. in recent years, sentiment analysis is increasingly being performed using deep learning approaches @xcite . @xcite designed a deep pyramid cnn which could efficiently represent long-range associations in text and thus more global information for better sentiment classification. @xcite have developed ulmfit, a simple efficient transfer learning method that achieves improvements for various nlp tasks such as sentiment classification. another model that performs well on sentiment classification is bert @xcite , where pre-trained language models can be fine-tuned without substantial effort to suit different tasks. @xcite showed that decreasing the learning rate layer-wise and further pre-training enhance the performance of bert. another approach from @xcite improves the performance of bert with the usage of data augmentation. it was shown that another current language model xlnet @xcite achieves the best results for the sentiment classification task. based on the idea of the same side stance classification task by @xcite as well as the authorship verification problem @xcite , our underlying hypothesis is that the more complex single sentiment problem may be able to be simplified to the semantic similarity of sentiment text pairs. this can then reduce the demand for topic-specific sentiment vocabulary usage @xcite . as there is no prior work about same sentiment classification, our work uses well-known approaches from semantic text similarity (sts) about which several shared tasks have been organized @xcite @xcite and a variety of datasets @xcite have been compiled. while prior approaches have employed syntactic, structural, and semantic similarity, to evaluate sentence similarity, single models have gained more popularity in recent times. @xcite show the application of siamese recurrent networks for sentence similarity. with the introduction of contextualized word embeddings, @xcite evaluate their impact on sts methods compared to traditional word embeddings in different languages and domains. ## the same sentiment problem in the following, we will introduce our model for same sentiment prediction and explain how to prepare training and test data. ## evaluation to thoroughly inspect our approach we conducted a series of experiments to test which hyperparameters are necessary to fine-tune a model in general, how well the model is able to generalize by artificially separating topics in training and evaluation, and how it performs for each category specifically. baseline as baseline models, we started with linear models, svm, and logistic regression classifiers, where we represented reviews as ngram count vectors, tf-idf word vectors, and as doc2vec @xcite embeddings. using count and tf-idf vectors, we were only able to achieve about 50% accuracy. with doc2vec embeddings, our accuracy improved to about 57%. those results most likely meant that those approaches were not a good fit for sentiment pair similarity prediction. we then used a siamese recurrent network architecture @xcite that has been successfully applied to semantic textual similarity problems. words were represented by pre-trained 50-dimensional glove @xcite embeddings. we set a maximum input sequence length of 256, 50 lstm cells in both bidirectional lstm layers and 50 hidden units. 7 training plateaued at 15 epochs with 83% accuracy. we will use the same configuration in all the following experiments. overall performance using bert, we started with an initial sequence length of 128, batch size of 32, and 5 epochs of fine-tuning but otherwise standard parameter choices to see how the model performs in general. the dataset consisted of 2 sentiment pairs for all 4 pair combinations for each business with a train/dev/test split of 90:10:10. this achieved 81.3% accuracy overall. increasing the sentiment pairs per business to 4 per type only increased accuracy to 82%, so the randomly chosen samples were enough to generally cover the dataset, and more samples only increased time per epoch but not significantly the result. we achieved a final 89.1% accuracy (cf. per sentiment pair type we further compared how a model trained on all sentiment pair types evaluates each type separately in ## conclusion our contribution in this paper is the introduction of a new perspective on sentiment analysis. we showed how sequence pair classification can be used to achieve relatively good accuracy on the same sentiment pair problem. initial results are promising but applying same sentiment models on