Robust Integration of Contextual Information for Cross-Target Stance Detection (2023)

Abstract: Stance detection deals with identifying an author’s stance towards a target. Most existing stance detection models are limited because they do not consider relevant contextual information which allows for inferring the stance correctly. Complementary context can be found in knowledge bases but integrating the context into pretrained language models is non-trivial due to the graph structure of standard knowledge bases. To overcome this, we explore an approach to integrate contextual information as text which allows for integrating contextual information from heterogeneous sources, such as structured knowledge sources and by prompting large language models. Our approach can outperform competitive baselines on a large and diverse stance detection benchmark in a cross-target setup, i.e. for targets unseen during training. We demonstrate that it is more robust to noisy context and can regularize for unwanted correlations between labels and target-specific vocabulary. Finally, it is independent of the pretrained language model in use.

## introduction given a text and a target the text is directed at, stance detection (sd) aims to predict whether the text contains a positive or negative stance towards the target or is unrelated. we provide an example in figure 1 . in contrast to formal polls, stance detection (sd) provides a scalable alternative to assess opinions expressed in unstructured texts. however, in contrast to predicting the polarity of a text (i.e., sentiment analysis), sd is more challenging because it requires establishing the relation towards a target which is rarely mentioned in the text @xcite . further, to infer the correct stance, often the text alone is not sufficient and contextual information needs to be taken into account @xcite . in contrast, most stance classification models are expected to make a correct prediction given the text and target only. this can lead to overly relying on label correlations with target-specific vocabulary @xcite . in our example (figure 1 ), it is challenging to follow the reasoning of the text if the meaning of school spirit is left unclear. target: school uniforms label: pro text: creates a sense of school spirit. context: ['school spirit is the enthusiasm and pride felt by the students of a school', 'a strong sense of school spirit is a positive and uplifting influence on the school and its students'] figure 1 : example for stance detection from the ukp argmin dataset @xcite . the context is not part of the original dataset and was extracted from a large language model via prompting. consequently, providing external knowledge as an additional signal to stance classification has been proposed as a remedy. however, lacking a general solution, previous work applies knowledge integration only for a specific text domain like social media @xcite . nevertheless, sd algorithms are applied on a multitude of different text sources like social media (ald), news @xcite or debating fora @xcite and on diverse targets such as persons @xcite , products @xcite , or controversial topics @xcite , among other things. in addition, existing approaches @xcite often depend on the structure of the external knowledge source used. however, a single source of knowledge will likely not suffice for all different scenarios and adapting the model architecture to the structure of a specific knowledge source (e.g. graph-based) limits its applicability. this work proposes a flexible and robust approach to integrate contextual information by encoding it as text. it is better aligned to the encoding schema of a pre-trained language model (plm) and circumvents any dependency on the structure of a particular knowledge source. it also allows for using any context source that best fits the data's text domain or mixing contextual information from multiple sources. in detail, we propose a dual-encoder architecture (inject), which encodes the input text and context information separately while facilitating information exchange between both via attention. we investigate extracting contextual information from various sources using different extraction strategies. we evaluate our approach across a benchmark of 16 stance detection datasets exhibiting different characteristics concerning text source, size, and label imbalance. first, we demonstrate that existing state-of-theart approaches outperform standard baselines only on the domains they have been tuned for -but perform worse on average. when integrating context via inject, we observe improvements on average and provide an analysis demonstrating the robustness of our approach. in summary, we make the following contributions: ## related work many tasks in nlp benefit from access to external knowledge such as natural language inference @xcite , machine translation @xcite or argument mining @xcite . within the era of plms, many approaches rely on extensive pretraining using data from knowledge bases @xcite (kb) or supervision from knowledge completion tasks @xcite . early works leveraged sentiment lexicons @xcite or combinations thereof @xcite to improve sd classification performance. other contextual components like author information @xcite @xcite , dissemination features of social media @xcite such as retweets or structural discourse elements @xcite have been shown to play an important role for stance detection. similarly to the aforementioned approaches, the focus in sd has shifted towards combining structural kbs and plms. kawintiranon and singh (2021) identify label-relevant tokens and prioritize those during masked language modeling. this approach risks overfitting on targetspecific tokens because stance is often expressed using target-specific terminology -an issue which is particularly problematic for argumentative sentences @xcite . @xcite apply a knowledge infusion method for plms by filtering wikipedia triplets for contextual knowledge. @xcite extend bert by introducing a consistency constraint to learn agreement between the text and its target. @xcite presents a variant of bert pre-trained using a variety of supervised tasks resembling logical mechanisms. @xcite extract relevant concepts from conceptnet using graph-based ranking methods to improve argument relation classification. likewise, @xcite uses con-ceptnet to identify relevant concept-edge pairs and integrate them via a graph neural network during training. finally, @xcite used label embeddings to improve sd multi-dataset learning and recently showed @xcite that sentiment-based pretraining improves multi-lingual stance detection. in summary, most existing approaches integrate knowledge through extensive pretraining on knowledge-rich data. this does not guarantee im-provement of the downstream task they are intended for and requires additional experiments. another line of work introduces architectural dependencies on the structure of the knowledge source, thereby limiting their usage to tasks and domains for which the knowledge source is applicable. in contrast, our approach does not require further pretraining but directly learns to integrate contextual information during supervised training. the usefulness of the context is, therefore, directly measurable. further, our proposed approach integrates context in natural language, thereby decoupling it from the structure of the context source. it is better aligned with the encoding mechanism of pre-trained language models and enables the integration of contextual information from various sources. ## methodology our goal is twofold: (1) we aim to integrate contextual information independent of the context source and (2) in a way that is robust to noisy and irrelevant content in the context. we propose inject, a dual encoder approach to integrate contextual sentences using the cross-attention mechanism introduced by @xcite . the general intuition is that information can flow from input to context and vice versa, thereby regularizing the attention in both encoders. thus, the context provides further information to reweigh the prediction importance of individual tokens in the input. ## results this section shows the effectiveness of inject by providing constant improvements using noisy context on the heterogeneous benchmark ( the baselines stancy, tga-net, and jointcl mostly show the best performance for the datasets they have been proposed for. however, on average, they do not perform on par with the strong bert+target baseline. stancy performs slightly worse, probably due to the binary contrastive loss and thereby ignoring multi-label information. tga-net underperforms all other approaches except for vast. using generalized topic representations transfers to a scenario where the number of targets is relatively high (5634) and only a few examples per target exist (mean 2.4), as for vast. jointcl performs best on datasets from social media (semeval16, wtwt, or mtsd), but is outperformed by standard baselines for the rest of the tasks. thus, this approach can not generalize to datasets with longer text inputs. we conclude that existing state-of-the-art approaches for cross-target stance detection work well for the datasets they have been designed for but do not generalize across the diverse set of datasets that exist in sd. inject outperforms bert+target in 13 of 16 cases, while for three datasets (perspectrum, poldeb, snopes) none of the extracted contexts provides benefits, independently of the integration. on average, all context sources lead to performance improvements in combination with inject, with t0pp-np-t being the best. combining all context sources underperforms the integration of individual context, most probably due to the average function leading to a perturbation of the context. surprisingly, integration of random context slightly outperforms the strong bert+target baseline in ten datasets while degrading the direct integration performance, as expected. we investigate the reasons in our subsequent analysis ( §6.3). ## conclusion we propose inject, a dual-encoder approach to integrate contextual information for stance detection based on cross-attention. while state-of-the-art approaches perform mostly well on the datasets they have been proposed for, we evaluate our approach across a large and diverse benchmark in a crosstarget setting and observe improvements compared using three different sources for extracting contextual information. we show that the context integrated via inject improves stance detection and is beneficial for generalization on targets not seen during training. in future, we plan to explore more sophisticated ways of prompting large pre-trained language models for helpful context.