Stanceformer: Target-Aware Transformer for Stance Detection (2024)

Abstract: The task of Stance Detection involves discerning the stance expressed in a text towards a specific subject or target. Prior works have relied on existing transformer models that lack the capability to prioritize targets effectively. Consequently, these models yield similar performance regardless of whether we utilize or disregard target information, undermining the task’s significance. To address this challenge, we introduce Stanceformer, a target-aware transformer model that incorporates enhanced attention towards the targets during both training and inference. Specifically, we design a Target Awareness matrix that increases the self-attention scores assigned to the targets. We demonstrate the efficacy of the Stanceformer with various BERT-based models, including state-of-the-art models and Large Language Models (LLMs), and evaluate its performance across three stance detection datasets, alongside a zero-shot dataset. Our approach Stanceformer not only provides superior performance but also generalizes even to other domains, such as Aspect-based Sentiment Analysis. We make the code publicly available.

## introduction stance detection is the task that involves determining the stance or perspective expressed in a given text towards a particular topic or target. the stance could be either favor, against, or neutral with respect to the target. it has many useful applications, including prediction of election/referendum results @xcite , analyzing online debates @xcite , social media analysis @xcite , rumor verification @xcite , rumor classification @xcite , automated fact-checking @xcite , fake news detection @xcite , and information retrieval @xcite . text: a woman ?? wanting to be equal to a man ???! what montrosity is this target: feminist movement stance: against targets play a pivotal role in the stance detection task because they define the context and subject matter of the stance expressed in a text. however, several studies @xcite have identified that the models tend to overlook the targets when making predictions. this challenges the definition of the stance detection task, which is inherently tied to a target (the author of a text takes a stance towards a specific target). therefore, it is crucial to ensure that models are aware of the targets when making decisions to improve their performance. to address these issues, researchers have proposed techniques such as data augmentation @xcite @xcite , leveraging external knowledge bases @xcite @xcite , employing human-like reasoning @xcite , and using contrastive learning-based methods @xcite to improve target representations during training. additionally, efforts have been made to enhance target-specific attention for bilstm-based models @xcite @xcite . however, we develop mechanism for enhancing target-specific attention in more sophisticated transformer models, as well as two state-of-the-art methods for stance detection @xcite . specifically, in this study, we introduce a novel target-specific attention mechanism designed for transformer models. the aim is to guide the models in incorporating additional attention to the specified target for each text. we add a special target-awareness (ta) matrix, which is incorporated into the self-attention matrix within a specific layer and head of the transformer model. the sparse ta matrix contains non-zero values only for the square block covering the target subwords (as illustrated in figure 2 ). experimental results demonstrate consistent improvements with our approach compared to the baseline models across various models and datasets, including both full-dataset and zero-shot dataset settings. our approach seamlessly integrates with any off-the-shelf stance detection model that uses transformer architectures. additionally, we apply our approach to various llms, including llama-2-chat models with 7 billion and 13 billion parameters, in both zero-shot inference and finetuned settings. our approach outperforms both bert-based language models and autoregressive llms. finally, we also showcase the generalization capabilities of our method in other domains, such as aspect-based sentiment analysis. we summarize our contributions as follows: target-awareness the stance detection task involves two key inputs: the text or tweet, and the target. numerous research efforts have focused on leveraging target information to enhance the performance of stance detection models. we classify these approaches into four main categories. data augmentation, open-domain/opentargets: @xcite introduced a novel dataset that emphasizes the role of target entities in stance detection. they achieved this by augmenting the dataset with samples that have negated (or unrelated) targets and opposite (or unrelated) stance labels. @xcite proposed an auxiliary sentence-based data augmentation technique that considers both the target and label, as well as target-specific replacement, to generate augmented samples. @xcite proposed to augment a dataset with additional targets to improve the stance detection task. the authors first augment the training set by extracting diverse targets (unseen during training) and then predict the stance by effectively utilizing the augmented data. @xcite argued that the targets for stance are not always available and proposed an approach called target-stance extraction that aims to extract (target, stance) pairs from the text. other works @xcite @xcite @xcite ) address open-domain/open-target for stance detection. ## proposed approach in this section, we describe our proposed method stanceformer, designed to enhance the target awareness of the transformer models. figure 2 provides a visual illustration of our proposed approach. in the following sections, we first formally define the stance detection task in §3.1, followed by a detailed explanation of the core component of stanceformer, namely, the target awareness matrix in §3.2. finally, we describe the process of finetuning with stanceformer in §3.3. ## experimental setup in this section, we first describe various stance detection datasets used for evaluation in §4.1, followed by an overview of various baseline models in §4.2. finally, we discuss the evaluation metrics used for different datasets in §4.3. ## results and discussion in this section, we provide a discussion on the results of the various models and stanceformer on three full dataset settings ( §5.1) and one zero-shot dataset setting ( §5.2). ## analysis we perform ablation experiments on the semeval-2016 dataset for the two models bert and bertweet and show the results in in ## generalization to aspect-based sentiment analysis domain aspect-based sentiment analysis (absa) focuses on identifying sentiment polarity (positive, negative, or neutral) in relation to specific aspect terms within a given text. this task closely aligns with stance detection (sd), as the aspects in absa resemble the targets in the sd task, making our approach particularly well-suited for application to absa. in this section, we evaluate the generalization capabilities of our approach on various benchmark datasets for absa, including semeval14 (laptops, restaurants), semeval15 (restaurants), and semeval16 (restaurants). we train bert and bertweet models, along with their stanceformer variants, on the four aspects-lapt14, rest14, rest15, and rest16-from the aforementioned datasets. the results are presented in tables 6 7 . key observations include that the stanceformer variants consistently outperform the base models (bert and bertweet) across all aspects. for the bertweet model, stanceformer improves macro-f1 by up to 4.1% for the rest16 aspect and accuracy by 1.4% for the lapt14 aspect. for the bert model, stanceformer enhances performance by up to 2.2% on macro-f1 and 3.2% on accuracy for the lapt14 aspect. the superior performance of stanceformer suggests that the models benefit from a more focused attention mechanism on the target (or aspect) terms, leading to better generalization. similar to the ablation study for the stability detection described in §6, we conduct an ablation study for the absa task, presented in ## conclusion existing transformer models lack the ability to effectively prioritize the targets involved in the stance detection task. in this work, we propose stanceformer, a modification to the transformer architecture specifically designed for stance detection, featuring enhanced target awareness. extensive experiments on four stance detection datasets using various transformer models, including bert-based models and autoregressive llms, demonstrate consistent performance improvements across all settings. additionally, we show that stanceformer generalizes well to other domains, such as aspect-based sentiment analysis.