Why Do Document-Level Polarity Classifiers Fail? (2021)

Abstract: Machine learning solutions are often criticized for the lack of explanation of their successes and failures. Understanding which instances are misclassified and why is essential to improve the learning process. This work helps to fill this gap by proposing a methodology to characterize, quantify and measure the impact of hard instances in the task of polarity classification of movie reviews. We characterize such instances into two categories: neutrality, where the text does not convey a clear polarity, and discrepancy, where the polarity of the text is the opposite of its true rating. We quantify the number of hard instances in polarity classification of movie reviews and provide empirical evidence about the need to pay attention to such problematic instances, as they are much harder to classify, for both machine and human classifiers. To the best of our knowledge, this is the first systematic analysis of the impact of hard instances in polarity detection from well-formed textual reviews.

## introduction document-level polarity classification is the task of classifying the polarity of a whole opinionated message @xcite . for instance, given a movie review, the system determines whether the review text expresses an overall positive, negative, or neutral opinion about the movie. although polarity classification naturally suits to analyze consumer opinions about products and services @xcite , it is also well suited to various types of applications, such as to infer votes in elections @xcite , civilian sentiment during terrorism scenarios @xcite , citizens' perception of government agencies @xcite and recommendation systems @xcite . supervised machine learning is one of the most common and successful approaches for polarity classification, but even state-of-the-art methods fail to correctly classify a substantial portion of the instances, from 10% to 20%, depending on the dataset @xcite . the problem with this approach is that if the data is not representative and reliable, the model is unlikely to perform well. one source of unreliability is data noise, which can be categorized into class noise and attribute noise @xcite . class noise occurs when the training data contains instances that are wrongly labeled. attribute noise occurs when the training data contains one or more attributes with wrong, incomplete or missing values. in the case of textual data, such noise usually comes in the form of errors in language rules, such as typos, grammatical errors, improper punctuation, and abbreviations @xcite @xcite . nevertheless, for both cases, the noise can be eliminated from the data by correcting the labels (for class noise) or the problematic text (for attribute noise). a more problematic source of data unreliability in polarity classification tasks comes from well written text that, for some reason, does not convey its class clearly. literature calls such instances hard instances, which are those that are intrinsically hard to correctly label or classify @xcite . differently from noisy instances, hard instances cannot be corrected, so the only solution is to identify and remove them from the training data. also, hard instances are not equivalent to outliers, as they do not differ significantly from other observations and may represent a significant portion of the data @xcite . for example, in a polarity classification task, a positive movie review that describes at least as many negative as positive points of the film can be a hard instance. to the best of our knowledge, no study exists that characterizes such instances and quantifies their impact on document-level polarity classification tasks. thus, we propose a methodology to characterize, quantify and measure the impact of hard instances in polarity classification tasks and demonstrate its usefulness in the task of movie review polarity classification. to this end, we collected 415, 867 positive and negative movie reviews from metacritic. one advantage of metacritic is that the meaning of ratings is clearly stated to the users when a review is being submitted: positive ratings range between 61% and 100%, neutral range between 40% and 60%, and negative between 0% and 39%. because of that, class noise and biases should be rare, that is, a user who liked (disliked) a movie will very unlikely give a negative (positive) rating to it. thus, classification errors will mostly be due to hard instances, which we assign into two disjoint categories: neutral and discrepant. a neutral review does not have a clear polarity and a discrepant review has a human-perceived polarity that is different from its associated rating. this categorization is complete, i.e., every instance that, for a human, does not reveal its class clearly falls into one (and only one) of these two types of hard instances. neutral and discrepant reviews are characterized by a well-defined human classifier that uses human reasoning to infer the class of the example. when the class assigned by the human classifier is incorrect, we label the review as discrepant, i.e., the human-perceived polarity of the text is different from its associated rating. when the human classifier is not confident about its prediction, we label the review as neutral. we labeled 1, 200 reviews and found 198 neutral and 64 discrepant reviews. we tested state-of-the-art machine classifiers on these reviews and results revealed that hard instances can significantly decrease their performances. in short, the main contributions are: as an additional contribution, we show how far are state-of-the-art machine classifiers from human performance in the task of movie review polarity classification. ## related work in supervised machine learning, class and attribute noise can increase learning complexity and, consequently, reduce classification accuracy @xcite . class noise is considered to be more harmful than attribute noise @xcite , but it is easier to detect @xcite . thus, class noise is more often addressed in the literature @xcite , where several studies analyzed its impact in classification tasks and how to address it @xcite @xcite @xcite . in nlp, attribute noise are unintended errors in text, which can come from failures in automatic character recognition processes @xcite or naturally while writing the text in the form of errors in language rules, such as typos, grammatical errors, improper punctuation, irrational capitalization and abbreviations @xcite @xcite @xcite . in short, noise are unintentional and undesirable errors in the text that can (and should) be eliminated from the data. conversely, hard instances are noise-free and cannot be corrected, only eliminated from the data @xcite . in addition, they differ from outliers because their feature representation vectors may be similar to others from regular instances @xcite . nevertheless, hard instances are more prone to class noise. in fact, @xcite defined hard instances in the context of label annotations, under the assumption that items that are easy are reliably annotated, whereas items that are hard display confusion and disagreement among the annotators. later, beigman klebanov and @xcite showed that the presence of hard instances in the training data misleads the machine learner on easy, clear-cut cases. the definition of @xcite is similar to ours: hard instances are simply those that "should be misclassified" by machine learning methods. the authors introduced hardness measures based on the outputs of an ensemble of classifiers to identify such instances and showed that classifiers are often uncertain about their classes. following the same idea, @xcite argues that easy instances are correctly classified by all or most classifiers. on the other hand, hard instances are missed by most of them. in this work, we propose a human classifier composed by human annotators to identify hard instances. our definition unifies the ones @xcite and @xcite . similarly to beigman klebanov and @xcite , we define hard instances as those in which the human classifier is uncertain or wrong about their true labels. however, different from these studies, which quantify the impact hard instances have on training, our goal is to provide a methodology to quantify the expected amount of hard instances in data and the impact they have on classifiers in production and testing. also, and similarly to @xcite , hard instances are divided into "instances that should be misclassified", which we call discrepant, and "border points", which we call neutral. to the best of our knowledge, we are the first to propose a methodology to characterize and quantify the impact of hard instances in unstructured textual data for polarity classification tasks. regarding the effect of hard instances in sentiment and polarity classification tasks, @xcite showed that it is easier to classify sentiment in short documents (e.g. tweets) than in longer ones, as short documents have less nonrelevant information. also, @xcite showed that ratings in tripadvisor reviews are not strongly correlated with sentiment scores given by sentiment analysis methods and proposed a unified index that aggregates both polarities. @xcite collected a subset of sentences that an ensemble of state-of-the-art sentiment classifiers misclassified and annotated them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, among others. in our work, we analyze manually identified hard instances (as opposed to instances misclassified by a machine classifier). as a result, compared to these works, we have a more precise (e.g., a misclassified instance is not necessarily hard) and complete (e.g., not all hard instances are misclassified) ground-truth. our methodology is composed by a human classifier f h , which identifies hard instances, and a machine classifier f m , which is tested on hard and regular instances. a classifier is defined as a function f (x i ) that receives a textual movie review x i as input and returns its polarity ŷi ∈ {0, 1}. we use the human classifier to assign a label l i to a large sample of movie reviews x i to indicate whether x i is a hard instance or not. this label can be one (and only one) of a set l of manually defined labels that indicate that the instance is regular or a type of hard instance. with that, we will be able to quantify the impact of hard instances on machine classifiers and provide explanations about why they occur and how to avoid them in order to improve machine classifiers' accuracy. more specifically, for a machine classifier f m and for all labels @xmath0, regular included, we will calculate the probabilities p (l @xmath1. ## conclusion in this work, we propose a methodology to characterize, quantify and measure the impact of hard instances in the task of polarity classification of movie reviews. we characterized such instances into two disjoint categories: neutrality and discrepancy. we provided empirical evidence about the need to pay attention to such instances, as they are much harder to be classified, for both machine and human classifiers. the main hypothesis of this work is that hard instances can make polarity classifiers fail. to demonstrate this hypothesis, we provided two well defined types of hard instances, which are based on human reasoning, and a methodology to find them in labeled data. with that, one can quantify how many instances of those types there are in their data, which can shed light on why and when classifiers fail. we collected a noise-free (no class noise) and well separated (no neutral polarity) dataset and showed that even in such a dataset most of the mistakes made by a state of the art classifier, namely bert, are in our defined hard instances. observe in our methodology works for every type of supervised classification task. because our proposed labels are defined from the perspective of a classifier fully capable of human-reasoning, they are easy to interpret and can be generalized to every classification task (e.g. polarity, image, song genre, topic) that humans are able to do. after employing our methodology, it will be possible to differentiate mistakes that come from hard instances, which are those even humans cannot classify with confidence (or at all), and mistakes that could be solved by improving the classifier architecture. in short, our proposed methodology can help quantify and explain why classifiers are making mistakes. we made the dataset containing the labels publicly available foot_2 so it can be used as a standard benchmark for robustness to hard instances in polarity classification tasks, and to potentially foster research on models, datasets and evaluation metrics tailored for this problem. examples "michael bay may think that special effects can substitute for good acting and a good story, but that does not fly around here." (negative) "from the first moment of superman till the very end scene lex luthor this is a true comic book movie adaption. true there are few cgi errors, but "nothing is perfect in this world" and this is just a movie." (positive) "the trailer was promising to me; i expected it to be a really good movie, but instead it was "meh". i didn't really like cruz; it was heartwarming how lightning mcqueen made a tribute to doc at the end, but the trailer made it seem action packed; it wasn't as good as i expected." (negative) we further investigate these patterns by using shap (lundberg and lee, 2017), which is a game theoretic approach to explain the output of deep learning models and designed to understand the most important words for the machine classifiers. figure 3 shows the result for the last review in