t WT – WT : A Dataset to Assert the Role of Target Entities for Detecting Stance of Tweets (2021)

Abstract: The stance detection task aims at detecting the stance of a tweet or a text for a target. These targets can be named entities or free-form sentences (claims). Though the task involves reasoning of the tweet with respect to a target, we find that it is possible to achieve high accuracy on several publicly available Twitter stance detection datasets without looking at the target sentence. Specifically, a simple tweet classification model achieved human-level performance on the WT–WT dataset and more than two-third accuracy on various other datasets. We investigate the existence of biases in such datasets to find the potential spurious correlations of sentiment-stance relations and lexical choice associated with the stance category. Furthermore, we propose a new large dataset free of such biases and demonstrate its aptness on the existing stance detection systems. Our empirical findings show much scope for research on the stance detection task and proposes several considerations for creating future stance detection datasets.

## introduction stance detection is a vital sub-task for fake news detection @xcite , automated fact checking @xcite , social media analysis @xcite , analyzing online debates @xcite and rumour verification @xcite . furthermore, it is also an essential measure for progress in natural language understanding, especially in the noisytext domain. over the recent years, several stance detection datasets have been proposed. these datasets, in turn, facilitated progress in stance detection research, with some systems achieving up to 93.7% accuracy @xcite . however, most of these state-of-the-art systems are complex deep neural networks, making them difficult to interpret. lack of explainability raises concern since previous works @xcite @xcite on other tasks demonstrated that superficial dataset biases could result in inflated test-set performance. with this motivation, we carry out the first study analyzing several publicly available twitter stance detection datasets. our experiments reveal rampant biases in datasets through which even target-oblivious models can achieve impressive performance. various existing works have hinted at the presence of such dataset biases. for example, tan model @xcite is a very competitive stance detection model.however, @xcite recently proved that tan does not take advantage of target information at all. in rumoureval-2017 @xcite , models delivered up to 0.74 accuracy without any knowledge of the target, being only short of 0.004 from the best model considering the context. similarly, in rumoureval-2019 @xcite , the runner-up model @xcite observed a 0.43 decrease in accuracy by considering the target information. @xcite discovered that stance detection models are prone to adversarial attacks of paraphrasing, spelling error and negation similar to other nlp tasks @xcite . however, this is the first work providing a detailed insight into the alarmingly impressive performance of target-oblivious models. target plays a crucial role in deciding stance. consider the example in figure 1 . here, the tweet stance varies for the two targets. the existing datasets have very few examples with different target labels. models can pick up on pseudo signals in the tweet content and shortcut the task without looking at the targets. these signals or biases occur due to inherent biases in our language and human nature. for example, certain lexical choices can correlate with their respective stance classes. upon discovering and studying such correlations, we augment the wt-wt dataset addressing these issues and re-evaluate the stance detection systems. we make the following contributions. we empirically demonstrate biases across a variety of twitter stance detection datasets and carry out a detailed analysis of these datasets. consequently, we propose a new large scale dataset free of such spurious cues and re-evaluate the stance detection systems to show the usefulness of this dataset. ## biases in stance detection datasets we first discuss the datasets considered (section §2.1), followed by our experiments (section §2.2) and analysis (section §2.3). ## the targeted wt-wt (twt-wt) dataset with the understanding from the previous section, we propose a new stance detection dataset on which target-unaware models will not perform well. we use the following reasoning for creating the new dataset. if a tweet in the dataset has different stances depending on different targets, then simple tweet classification models will not be able to perform well. thus we attempt to increase dt/t ratio from second, we remove lexicon-stance correlations by creating multiple targets with different stances for each tweet. formally, for each tweet t with only one labelled target tgt, if the tweet-target pair (t, tgt) has the @xmath0, then pick a target tgt where @xmath1, tgt , u nrelated) to the dataset. due to wt-wt data collection and annotation procedure, this will not generate any wrong labels. this augmentation reduces the lexicon-stance correlations, by having similar sets of lexicons introduced for different stances. hence, it guarantees target-oblivious shortcuts to result in poor performance. last, we balance the target-wise classdistributions. for the tuples with 'comment' and 'unrelated' stances, we create a new tuple with inverted target (same as the first step) for 50% and 75% such examples randomly. the resulting dataset contains 111596 tweettarget pairs each belonging to a stance class. each merger has at least 10000 data points. the class distribution is also somewhat balanced with more than 10k examples for the least occurring class. among the tweet-target pairs, the pairs classified as support, refute, comment and unrelated are distributed in the ratio 1:1:3:5 approximately, having a similar distribution to the wt-wt dataset. ## conclusion and future work in this paper we demonstrated the presence of biases across several twitter stance detection datasets, which aid simple tweet classifiers to achieve impressive performance. we carried out an investigation for presence of bias for the wt-wt dataset and found correlations of stance-class with sentiment and lexical choice. consequently, we proposed a new bias-free stance detection dataset -twt-wt, the largest of its kind. evaluation of our baselines on this new dataset demonstrates scope for future research on stance detection. the observations are also crucial for the creation of new stance detection datasets. our future work includes analysing multilingual datasets and exploring explainable target aware stance detection models.