Integrating Transformers and Knowledge Graphs for T witter Stance Detection (2021)

Abstract: Stance detection (SD) entails classifying the sentiment of a text towards a given target, and is a relevant sub-task for opinion mining and social media analysis. Recent works have explored knowledge infusion supplementing the linguistic competence and latent knowledge of large pre-trained language models with structured knowledge graphs (KGs), yet few works have applied such methods to the SD task. In this work, we first perform stance-relevant knowledge probing on Transformers-based pre-trained models in a zero-shot setting, showing these models’ latent real-world knowledge about SD targets and their sensitivity to context. We then train and evaluate new knowledge-enriched stance detection models on two Twitter stance datasets, achieving state-of-the-art performance on both.

## introduction stance detection (sd) involves identifying a text's stance towards a given target (for example, whether a tweet is supportive, against, or neutral towards joe biden). this is a challenging task with downstream use cases in opinion mining, fake news detection, and rumor verification @xcite @xcite . a major challenge for sd is the need for knowledge of current events and other fastchanging facts about the world. large pre-trained transformer models trained on vast corpora @xcite have blurred the line between language models and knowledge bases, as shown by their performance on benchmarks like lama, which measure static factual knowledge @xcite . recent works in sd have capitalized on the transformer architecture; however, it remains uncertain how to adapt these models to the constantly shifting factual landscape found in sd tasks, for example in political tweets. at the same time, knowledge infusion (ki) approaches have had success in integrating kgs with transformers for question-answering (qa) tasks, but there is a shortage of work on ki for sd. our contributions are as follows: 1) we perform stance-relevant knowledge probing on transformers-based pre-trained models, showing these models' partial real-world knowledge and sensitivity to context, and 2) we train and evaluate knowledge-enriched stance detection models on two twitter stance datasets, achieving state-of-theart performance on both. ## previous work the original author baseline for the semeval-16 sd task used an svm classifier on hand-crafted features @xcite . more recent approaches for sd have achieved better performance using transfer learning with transformer models, but without adding knowledge infusion @xcite @xcite . this typically involves concatenating a tweet and target and feeding it into a transformer model with a classification layer attached. to our knowledge, @xcite is the only sd work using "knowledge enhancement", but this approach was based on identifying stancesignaling words rather than using kgs. most attempts to augment transformer models with structured knowledge from kgs have focused on qa tasks, such as commonsenseqa @xcite , not sd. @xcite used con-ceptnet @xcite to extract knowledge descriptions relating entities in each question to entities in each answer choice via multi-hop reasoning on a kg, with a bert-based classifier to choose the best answer. similarly, k-bert @xcite enriches entities in an input sentence based on lookup in a kg. the success of these methods suggests that downstream tasks can benefit from contextual priming, where the same input supplemented with additional factual context leads to better predictions. this can be contrasted with approaches like k-adapter and knowbert, which infuse knowledge by modifying the model architecture, rather than by adding context to the input @xcite . one advantage of contextual priming is the ability to leave a model's architecture largely unchanged, requiring only a method of collecting and generating useful factual context. ## probing transformers for stance-relevant knowledge in this section, we seek to establish a lower bound for the stance-relevant knowledge already present in transformer models before doing any knowledge infusion. rather than testing recall of encyclopedic facts, we probe whether models make stance-related inferences regarding real-world entities in a human-like way. the three models we test are roberta-base @xcite , roberta-large, and twitter-roberta @xcite . roberta is pre-trained on a large internet corpus, including news articles, while twitter-roberta is trained on ∼ 58m tweets, making these models good candidates for political sd on twitter. using each model in a masked language modeling setting, we feed it a sentence with a single word replaced by the special [mask] token, returning a probability distribution over all vocabulary tokens. rather than using an automatically-generated knowledge benchmark like lama @xcite , we draw on a human-in-the-loop paradigm @xcite and manually design probes relevant to the sd task. examples involve public figures and political issues that appear as targets in major sd datasets @xcite , such as donald trump, hillary clinton, and climate change. we evaluate the models using accuracy@1, which is the percentage of prompts for which the highest-probability token generated by a model is appropriate in context, e.g. factually correct or aligned with reasonable inferences. some key example pairs are shown below: in example (1) above, all models were able to predict "boys" for the masked token. in example (2), all models predicted "zimmerman" rather than the desired "floyd"; this illustrates how models can quickly become outdated when new words and names enter common usage after a model is pretrained. save the earth! (4) i think that climate change is such a [mask]. ## experiments we consider two twitter datasets for sd: semeval-16 @xcite and @xcite . the first involves a range of controversial political targets, such as abortion, atheism, and the 2016 u.s. presidential election. the task is to predict a class label from among {favor, against, neither}. the dataset contains 2914 training examples and 1249 test examples. the second centers exclusively on the 2020 u.s. presidential election, and the prediction classes are against, favor, neither, neutral, and mixed. the dataset contains 2400 training examples and 600 test examples. most sd models in the literature use semeval-16 as a benchmark, and recent works have used bert to achieve new state-of-the-art performance @xcite @xcite . our general architecture for sd with transformers involves the target (plus optional knowledge enrichments) and the tweet being concatenated with an intervening separator token before being fed into a roberta model with a classification head. the weights of the entire network are updated during training. the architecture is very similar to that used in other transformer-based sd models @xcite @xcite . we compare our knowledge infusion models with base models fine-tuned on the same task data, as well as with k-adapter, described in section 2 @xcite . as reported in semeval-16 grimminger & klinger model acc. f1 ag. fav. nei. neu. baseline* -69.1 79.0 89.0 95.0 53.0 roberta-base 71.8 71.4 81.8 91.4 93.9 60.0 twitter-roberta 71.4 71.7 82.7 90.0 94.5 56.4 k-adapter 74.5 74.8 86.1 93.2 94.1 63.8 roberta-large 76.9 77.3 86.9 92.2 93.6 62.7 + entities 77.2 78.5 86.9 92.9 94.6 65.1 + paths 75.7 76.1 86.8 92.5 95.2 68.3 + ect 75.1 76.2 87.0 93.7 96.0 67.2 while there were also examples where the generated knowledge descriptions were irrelevant or noisy, these examples demonstrate how an appropriate knowledge description can improve downstream model performance. examples ( 9 ) and ( 11 ) illustrate the utility of multi-hop reasoning, adding context relating donald trump to joe biden and the color red, respectively. example (10) illustrates the usefulness of backing off to simple entityenrichment in cases where no knowledge paths exist, providing useful additional context about the company regeneron. ## conclusion in this work, we highlighted three key points based on knowledge probing: transformer models contain latent stance-relevant knowledge, roberta-large is better at this than the other models, and models can be misled by sentence context. we also established new state-of-the-art performance on two sd datasets using knowledge infusion. we introduce a novel method, edge cost tuning, that uses training data to re-weight the connections in a knowledge graph, which produced best results on one of the two sd tasks. our approach depends greatly on choice of kg and edge cost weighting method, so future work can explore additional ways of filtering for informative edges in a kg.