Will-They-Won’t-They: A Very Large Dataset for Stance Detection on T witter (2020)

Abstract: We present a new challenging stance detection dataset, called Will-They-Won’t-They (WT–WT), which contains 51,284 tweets in English, making it by far the largest available dataset of the type. All the annotations are carried out by experts; therefore, the dataset constitutes a high-quality and reliable benchmark for future research in stance detection. Our experiments with a wide range of recent state-of-the-art stance detection systems show that the dataset poses a strong challenge to existing models in this domain.

## introduction apart from constituting an interesting task on its own, stance detection has been identified as a crucial sub-step towards many other nlp tasks @xcite . in fact, stance detection is the core component of fake news detection @xcite , fact-checking @xcite , and rumor verification @xcite . despite its importance, stance detection suffers from the lack of a large dataset which would allow for reliable comparison between models. we aim at filling this gap by presenting will-they-won't-they (wt-wt), a large dataset of english tweets targeted at stance detection for the rumor verification task. we constructed the dataset based on tweets, since twitter is a highly relevant platform for rumour verification, which is popular with the public as well as politicians and enterprises @xcite . to make the dataset representative of a realistic scenario, we opted for a real-world application of the rumor verification task in finance. specifically, we constructed the dataset based on tweets that discuss mergers and acquisition (m&a) operations between companies. m&a is a general term that refers to various types of financial transactions in which the ownership of companies are transferred. an m&a process has many stages that range from informal talks to the closing of the deal. the discussions between companies are usually not publicly disclosed during the early stages of the process @xcite . in this sense, the analysis of the evolution of opinions and concerns expressed by users about a possible m&a deal, from its early stage to its closing (or its rejection) stage, is a process similar to rumor verification @xcite . moreover, despite the wide interest, most research in the intersection of nlp and finance has so far focused on sentiment analysis, text mining and thesauri/taxonomy generation @xcite @xcite . while sentiment @xcite and targetedsentiment analysis @xcite have an undisputed importance for analyzing financial markets, research in stance detection takes on a crucial role: in fact, being able to model the market's perception of the merger might ultimately contribute to explaining stock price re-valuation. we make the following three contributions. firstly, we construct and release wt-wt, a large, expert-annotated twitter stance detection dataset. with its 51,284 tweets, the dataset is an order of magnitude larger than any other stance detection dataset of user-generated data, and could be used to train and robustly compare neural models. to our knowledge, this is the first resource for stance in the financial domain. secondly, we demonstrate the utility of the wt-wt dataset by evaluating 11 competitive and state-of-the-art stance detection models on our benchmark. thirdly, we annotate a further m&a operation in the entertainment domain; we investigate the robustness of best-performing models on this operation, and show that such systems struggle even over small domain shifts. the entire dataset is released to enable research in stance detection and domain adaptation. ## building the wt-wt dataset we consider five recent operations, 4 in the healthcare and 1 in the entertainment industry ( ## experiments and results we re-implement 11 architectures recently proposed for stance detection. each system takes as input a tweet and the related target, represented as a string with the two considered companies. a detailed description of the models, with references to the original papers, can be found in appendix b.1. each architecture produces a single vector representation h for each input sample. given h, we predict ŷ with a softmax operation over the 4 considered labels. ## conclusions we presented wt-wt, a large expert-annotated dataset for stance detection with over 50k labeled tweets. our experiments with 11 strong models indicated a consistent (>10%) performance gap between the state-of-the-art and human upperbound, which proves that wt-wt constitutes a strong challenge for current models. future research directions might explore the usage of transformer-based models, as well as of models which exploit not only linguistic but also network features, which have been proven to work well for existing stance detection datasets @xcite . also, the multi-domain nature of the dataset enables future research in cross-target and crossdomain adaptation, a clear weak point of current models according to our evaluations. label 1: support -if the tweet is supporting the theory that the merger is happening. supporting tweets can be, for example, one of the following: 1. explicitly stating that the deal is happening: