Knowledge Enhanced Masked Language Model for Stance Detection (2021)

Abstract: Detecting stance on Twitter is especially challenging because of the short length of each tweet, the continuous coinage of new terminology and hashtags, and the deviation of sentence structure from standard prose. Fine-tuned language models using large-scale in-domain data have been shown to be the new state-of-the-art for many NLP tasks, including stance detection. In this paper, we propose a novel BERT-based fine-tuning method that enhances the masked language model for stance detection. Instead of random token masking, we propose using a weighted log-odds-ratio to identify words with high stance distinguishability and then model an attention mechanism that focuses on these words. We show that our proposed approach outperforms the state of the art for stance detection on Twitter data about the 2020 US Presidential election.

## introduction stance detection refers to the task of classifying a piece of text as either being in support, opposition, or neutral towards a given target. while this type of labeling is useful for a wide range of opinion research, it is particularly important for understanding the public's perception of given targets, for example, candidates during an election. for this reason, our focus in this paper is on detecting stance towards political entities, namely joe biden and donald trump during the 2020 us presidential election. stance detection is related to, but distinct from the task of sentiment analysis, which aims to extract whether the general tone of a piece of text is positive, negative, or neutral. sobhani and colleagues @xcite show that measures of stance and sentiment are only 60% correlated. for example, the following sample tweet 1 has an 1 all of the sample tweets in this paper are invented by the authors. they are representative of real data, but do not obvious positive sentiment, but an opposing stance towards donald trump. i'm so happy biden beat trump in the debate. stance detection is an especially difficult problem on twitter. a large part of this difficulty comes from the fact that twitter content is short, highly dynamic, continually generating new hashtags and abbreviations, and deviates from standard prose sentence structure. recently, learning models using pre-training @xcite @xcite have shown a strong ability to learn semantic representation and outperform many state-of-the-art approaches across different natural language processing (nlp) tasks. this is also true for stance detection. the strongest models for stance detection on twitter use pre-trained bert @xcite . a recent study that proposed models for sentiment analysis @xcite showed that focusing the learning model on some relevant words, i.e. sentiment words extracted using pointwise mutual information (pmi) @xcite , performed better than using the standard pre-trained bert model. we are interested in understanding whether or not focusing attention on specific stance-relevant vocabulary during the learning process will improve stance detection. to accomplish this, we consider the following two questions. first, how do we identify the most important stance-relevant words within a data set? and second, how much attention needs to be paid to these words versus random domain words? toward that end, we propose building different knowledge enhanced learning models that integrate an understanding of important contextspecific stance words into the pre-training process. while we consider pmi as a way to identify important stance words, we find that using the log-odds ratio performs better. we also consider different options for fine-tuning an attention-based language model. to fine-tune an attention-based language model to a specific task, the most common approach is to fine-tune using unlabeled data with random masking @xcite . because of the noise within social media posts, random tokens that are not taskrelevant can impact sentence representation negatively. therefore, instead of letting the model pay attention to random tokens, we introduce knowledge enhanced masked language modeling (ke-mlm), where significant tokens generated using the log-odds ratio are incorporated into the learning process and used to improve a downstream classification task. to the best of our knowledge, this is the first work that identifies significant tokens using log-odds-ratio for a specific task and integrates those tokens into an attention-based learning process for better classification performance. in summary, we study stance detection on english tweets and our contributions are as follows. ## related work in the nlp community, sentiment analysis is a more established task that has received more attention than stance detection. a sub-domain of sentiment analysis is target-directed or aspect-specific sentiment, which refers to the tone with which an author writes about a specific target/entity or an aspect of a target @xcite . one common use case is breaking down sentiment toward different aspects of a product in reviews, e.g., the price of a laptop versus its cpu performance @xcite @xcite . different approaches have been proposed to tackle this problem. chen and colleagues combine attention with recurrent neural networks @xcite . schmitt and colleagues propose combining a convolutional neural network and fasttext embeddings @xcite . a recent study proposes modifying the learning objective of the masked language model to pay attention to a specific set of sentiment words extracted by pmi @xcite . the model achieves new state-ofthe-art results on most of the test data sets. because stance is a different task, foot_1 we will adjust their target-directed sentiment approach for stance and compare to it in our empirical evaluation. the most well-known data for political stance detection is published by the semeval 2016 @xcite . the paper describing the data set provides a highlevel review of approaches to stance detection using twitter data. the best user-submitted system was a neural classifier from mitre @xcite which utilized a pre-trained language model on a large amount of unlabeled data. an important contribution of this study was using pretrained word embeddings from an auxiliary task where a language model was trained to predict a missing hashtag from a given tweet. the runner-up model was a convolutional neural network for text classification @xcite . following the mitre model, there were a number of both traditional and neural models proposed for stance detection. a study focusing on traditional classifiers proposed using a support vector machine (svm) with lexicon-based features, sentiment features and textual entailment feature @xcite . another svm-based model consisted of two-step svms @xcite . in the first step, the model predicts whether an input sequence is relevant to a given target. the next step detects the stance if the input sequence is relevant. target-specific attention neural network (tan) is a novel bidirectional lstm-based attention model. in this study, dey and colleagues trained it on unpublished unlabeled data to learn the domain context @xcite . recently, a neural ensemble model consisting of bi-lstm, nested lstms, and an attention model was proposed for stance detection on twitter @xcite . the model's embedding weights were initialized with the pre-trained embeddings from fasttext @xcite . the emergence of transformer-based deep learning models has led to high levels of improvement for many nlp tasks, including stance detection @xcite @xcite . bert @xcite is the most used deep transformer encoder. more specifically, bert uses masked language modeling (mlm) to pre-train a transformer encoder by predicting masked tokens in order to learn the semantic representation of a corpus. ghosh and colleagues @xcite show that the original pre-trained bert without any further finetuning outperforms other former state-of-the-art models on the semeval set including the model that utilizes both text and user information @xcite . because we are interested in the 2020 us presidential election and many temporal factors relevant to stance exist (e.g. political topics), we introduce a new election 2020 data set. for our empirical analysis, we will use this data set, and compare our approach to other state-of-the-art methods that used the semeval data set. our data sets are described in section 5.1. inspired by bert, different variations of bert have been proposed to solve different specific nlp tasks. spanbert @xcite masks tokens within a given span range. ernie @xcite finds and masks entity tokens achieving new state-of-the-art results on many chinese nlp tasks, including sentiment analysis. glossbert @xcite uses gloss knowledge (sense definition) to improve performance on a word sense disambiguation task. sensebert @xcite aims to predict both masked words and the word-net super-sense to improve word-in-context tasks. zhang and colleagues introduce entity token masking @xcite for relation classification where the goal is to classify relation labels of given entity pairs based on context. a number of studies have been working on adjusting transformers for sentiment analysis tasks. a recent study @xcite proposes a sentiment knowledge enhanced pre-training method (skep). it shows that masking sentiment words extracted by pmi guides the language model to learn more sentiment knowl-edge resulting in better sentiment classification performance. sentilare @xcite uses an alternative approach that injects word-level linguistic knowledge, including part-of-speech tags and sentiment polarity scores obtained by sentiword-net @xcite , into the pre-training process. following these works, sentix @xcite was proposed to incorporate domaininvariant sentiment knowledge for cross-domain sentiment data sets. our work differs because our task is stance detection and we employ a novel knowledge mining step that uses log-odds-ratio to determine significant tokens that need to be masked. ## ke-mlm: knowledge enhanced masked language modeling we propose knowledge enhanced masked language modeling (ke-mlm), which integrates knowledge that enhances the classification task in the fine-tuning process. we identify task-relevant tokens using text mining (section 3.1). we then use these discovered tokens within a masked language model (section 3.2). ## experimental design in this section we describe our experimental design, beginning with the knowledge mining decisions, followed by the decisions and parameters used for the language models. ## empirical evaluation after describing our data set (section 5.1), we present our experimental evaluation, both quantitative (section 5.2), and qualitative (section 5.3). ## conclusions and future directions intuitively, a language model fine-tuned using indomain unlabeled data should result in better classification performance than using the vanilla pretrained bert. since our goal is to maximize the accuracy of a specific classification task, we train an attention-based language model to pay attention to words that help distinguish between the classes. we have shown that for stance detection, using the log-odds-ratio to identify significant tokens that separate the classes is important knowledge for this classification task. once these important tokens are identified, forcing the language model to pay attention to these tokens further improves the performance when compared to using standard data for fine-tuning. to the best of our knowledge, our approach is better than the other state-of-the-art approaches for stance detection. additionally, we are releasing our data set to the community to help other researchers continue to make progress on the stance detection task. we believe this is the first stance-labeled twitter data for the 2020 us presidential election. there are several future directions of this work. first, to relax the trade-off between learning election semantics in general and learning stance knowledge, instead of fine-tuning one epoch with the normal fine-tuning method and another epoch with ke-mlm, we could reduce the masking probability of stance distinguishing words from 100% to something lower based on the distinguishability of the token. theoretically, this would give a higher weight to words that are more polarizing. this also relaxes the potential overfitting that may occur when learning only stance knowledge and lets the model randomly learn more tokens. another future direction is to test our language modeling method on other classification tasks (e.g. sentiment analysis, spam detection). also, this paper uses bert as the base language model. there are many variations of bert that can be further investigated (e.g. roberta). finally, we view stance as an important task for understanding public opinion. as our models get stronger, using them to gain insight into public opinion on issues of the day is another important future direction.