Tribrid: Stance Classification with Neural Inconsistency Detection (2021)

Abstract: We study the problem of performing automatic stance classification on social media with neural architectures such as BERT. Although these architectures deliver impressive results, their level is not yet comparable to the one of humans and they might produce errors that have a significant impact on the downstream task (e.g., fact-checking). To improve the performance, we present a new neural architecture where the input also includes automatically generated negated perspectives over a given claim. The model is jointly learned to make simultaneously multiple predictions, which can be used either to improve the classification of the original perspective or to filter out doubtful predictions. In the first case, we propose a weakly supervised method for combining the predictions into a final one. In the second case, we show that using the confidence scores to remove doubtful predictions allows our method to achieve human-like performance over the retained information, which is still a sizable part of the original input.

## introduction the spreading of unverified claims on social media is an important problem that affects our society at multiple levels @xcite @xcite . a valuable asset that we can exploit to fight this problem is the set of perspectives that people publish about such claims. these perspectives reveal the users' stance and this information can be used to help an automated framework to determine more accurately the veracity of rumors @xcite . the stance can be generally categorized either as supportive or opposing. for instance, consider the claim "the elections workers in wisconsin illegally altered absentee ballot envelopes". a tweet with a supportive stance is "the number of people who took part in the election in wisconsin exceeded the total number of registered voters" while one with an opposed stance is "an extra zero was added as votes accidentally but it was quickly fixed after state officials noticed it". being able to automatically classify the stance is necessary for dealing with the large volume of data that flows through social networks. to this end, earlier solutions relied on linguistic features, such as ngrams, opinion lexicons, and sentiment @xcite @xcite while more recent methods additionally include features that we can extract from networks like twitter @xcite @xcite . the current state-ofthe-art relies on advanced neural architectures such as bert @xcite and returns remarkable performance. for instance, stancy @xcite can achieve an f 1 of 77.76 with the perspectrum (per) dataset against the f 1 of 90.90 achieved by humans @xcite . although these results are encouraging, the performance has not yet reached a level that it can be safely applied in contexts where errors must be avoided at all costs. consider, for instance, the cases when errors lead to a misclassification of news about a catastrophic event, or when they trigger wrong financial operations. in such contexts, we argue that it is better that the ai abstains from returning a prediction unless it is very confident about it. this requirement clashes with the design of current solutions, which are meant to "blindly" make a prediction for any input. therefore, we see a gap between the capabilities of the state-of-the-art and the needs of some realistic use cases. in this paper, we address this problem with a new bert-based neural network, which we call tribrid (triplet bert-based inconsistency detection), that is designed not only to produce a reliable and accurate classification, but also to test its confidence. this test is implemented by including a "negated" version of the original perspective as part of the input, following the intuition that a prediction is more trustworthy if the model produces the opposite outcome with the negated perspective. if that is not the case, then the model is inconsistent and the prediction should be discarded. testing the consistency of the model with negated perspectives is a task that can be done simply by computing two independent predictions, one with the original perspective and one with the negated one. however, this is suboptimal because existing state-of-the-art methods are trained only with the principle that supportive perspectives should be similar to their respecting claims in the latent space @xcite and the similarity might be sufficiently high even if there are keywords (e.g., "not") that negate the stance. to overcome this problem, we propose a new neural architecture that processes simultaneously both original and negated perspectives, using a siamese bert model and a loss function that maximises the distance between the two perspectives. in this way, the model learns to distinguish more clearly supportive and opposite perspectives. to cope with the large volume of information that flows through social networks, it is important that the negated perspectives are generated automatically or at least with a minimal human intervention. to this end, two types of techniques have been presented in the literature. one consists of attaching a fixed phrase which negates the meaning @xcite while the other adds or removes the first occurrences of tokens like "not" @xcite . tribrid implements this task in a different way, namely using simple templates that negate both with keywords (e.g., "not") and antonyms. the prediction scores obtained with tribrid can be used either to determine more accurately the stance of the original perspective or to discard lowquality predictions. we consider both cases: in the first one, we propose an approach where multiple classifiers are constructed from the scores and a final weakly supervision model combines them. in the second one, we propose several approaches to establish the confidence and describe how to use them to discard low-quality predictions. our experiments show that our method is competitive in both cases. for instance, in the second case, our approach was able to achieve a f 1 of 87.43 on per by excluding only 39.34% of the perspectives. moreover, the score increased to 91.26 when 30% more is excluded. such performance is very close to the one of humans and this opens the door to an application in contexts where errors are very costly. the source code and other experimental data can be found at https://github.com/ karmaresearch/tribrid . ealier methods for stance classification employed various traditional classifiers, which include rule-based algorithms @xcite ; supervised classifiers like svm (hasan and ng, 2013), naïve bayes @xcite , boosting @xcite , decision tree and random forest @xcite , hidden markov models (hmm) and conditional random fields @xcite ; graph algorithms such as maxcut @xcite , and other approaches such as integer linear programming @xcite and probabilistic soft logic @xcite . popular features include cue/topic words, argumentrelated and sentiment/subjectivity features, and frame-semantic features. other features like tweets reply, rebuttal information, and retweets are known to improve the performance @xcite . in more recent work, the nlp community investigated on how to use deep neural network to improve the performance. some representatives are lstm-based approaches @xcite @xcite , rnn-based approaches @xcite , cnn approaches @xcite [sep] the generation of negated perspectives can be viewed as an instance of the broader problem of constructing adversarial examples, which is drawing more attention in nlp research in recent years @xcite . the main focus of adversarial generation is to change the text (with character/words changes or removals) to train more robust models. most of the works focus on changes that do not alter the semantics of the original input @xcite @xcite while works that negate the semantics are many fewer. in this context, some initial works have manually constructed some small-scale tests @xcite . more recently, gardner et al. ( 2020 ) suggested that datasets should be perturbed by experts with small changes to the test instances. in this way, empirical evaluations can test more accurately the true linguistic capabilities of the models. in their evaluation, they selected per, one of the datasets that we also consider, and showed that models such as bert perform significantly worse on the perturbed dataset. also ribeiro et al. (2020) consider the problem that accuracy on a held-out dataset may overestimate the performance on a real scenario. to counter this, they propose a new methodology that involves tests with certain perturbations (like negation), but they do not consider stance classification. these works further motivate our effort to develop models that are more consistent when presented with negated inputs. moreover, another goal of our work is to use consistency as a proxy to measure uncertainty and to discard low-quality predictions. a similar objective was pursued by @xcite for the problem of rumor verification. since manually creating additional test instances is time consuming, some works propose automatic procedures to generate them. @xcite proposes to add a fixed phrase at the end ("but this is not true") while niu and bansal (2018) adds the token "not" before the first verb in the sentence or replaces it with its antonym. finally, @xcite suggests a simpler alternative to remove the first occurrence of "not". in contrast to them, we use templates in the form of if-then rules. ## our approach first, we provide a short description on how bert has been used to achieve the state-of-the-art for this problem (section 3.1). then, we describe our proposed neural network (section 3.2) and how we can interpret its output to classify the stance (section 3.3). finally, we discuss the generation of negated perspectives using templates (section 3.4). ## evaluation we tested our approach on the datasets per @xcite and ibmcs @xcite , which are the main datasets previously used by our competitors. per is a set of claims and perspectives constructed from online debate websites while ibmcs is a similar dataset released by ibm. statistics on both datasets are in we implemented our approach using pytorch 1.4.0, and a bert base model with 12 layers, 768 hidden size, and 12 attention heads. we finetuned bert with grid search optimizing the f 1 on a validation dataset with a learning rates {1, 2, 3, 4, 5} × 10 -5 , batch size {24, 28, 32}, and the adam optimizer. for our experiments, we used a machine with a titanx gpu with 12gb ram. in the following, we first discuss the results using the confidence based k τ and λ τ classifiers (tribrid l and tribrid d ). then, we study the performance with our weakly supervised approach. finally, we provide an analysis on the coverage of the templates for negating the perspectives. ## conclusion in this paper, we introduced a new method to classify the stance of messages about a given claim. the main idea is to "inject" negated perspectives that are automatically generated into a bert-based model so that we can filter out dubious predictions or to improve the overall accuracy. if we filter out dubious predictions, then we can improve the performance to a point where the f 1 reaches a human-like level without sacrificing a large part of the input. we believe that discarding a (small) percentage of the input is not a major issue in data-intensive environments (e.g., social networks) where many users express their perspectives. however, if the use case is such that we must always make a prediction, then we have shown how we can leverage weak supervision to make a judicious prediction based on the confidence of the model. also this approach is competitive against the state-of-the-art on standard benchmark datasets. our work opens the door to several followup studies. a natural continuation is to explore whether we can achieve similar results if we negate the claim instead of the perspective. moreover, it is interesting to see whether we can construct paraphrases instead of negated text and modify the architecture accordingly. if we increase the number of sequences that we pass as inputs, our "siamese" approach may no longer work. if this occurs, then future work is needed to find some alternatives. finally, more sophisticated ways to negate the text may lead to further improvements. in general, we believe that critically assessing the output of a bert model using negated text is a promising technique to evaluate the model's confidence. therefore, it can also bring some improvements in other tasks like sentiment analysis, entity linking, or word sense disambiguation.