STANDER : An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval (2020)

Abstract: We present a new challenging news dataset that targets both stance detection (SD) and fine-grained evidence retrieval (ER). With its 3,291 expert-annotated articles, the dataset constitutes a high-quality benchmark for future research in SD and multi-task learning. We provide a detailed description of the corpus collection methodology and carry out an extensive analysis on the sources of disagreement between annotators, observing a correlation between their disagreement and the diffusion of uncertainty around a target in the real world. Our experiments show that the dataset poses a strong challenge to recent state-of-the-art models. Notably, our dataset aligns with an existing Twitter SD dataset: their union thus addresses a key shortcoming of previous works, by providing the first dedicated resource to study multi-genre SD as well as the interplay of signals from social media and news sources in rumour verification.

## introduction starting from early work by @xcite , stance detection (sd) has gained increasing interest from the research community @xcite . recent work in sd has mostly focused on modeling user-generated data @xcite ). however, sd on complex and articulated texts, such as news articles, has been considerably less studied, mainly due to the scarcity of published datasets @xcite . moreover, research on user-generated sd and news sd has proceeded on parallel and independent tracks, neglecting the deep mutual influence that exists between social media and news sources @xcite . in this paper, we seek to fill this gap, introducing stander (stance detection & evidence retrieval), a new expert-annotated dataset which is labeled for both news sd and fine-grained er. stander collects news articles in english from high-reputation sources which discuss four recent mergers and acquisitions (m&a) operations between major healthcare companies in the us ( notably, the news articles in stander discuss the same targets as in wt-wt @xcite , a large twitter sd dataset: thus, their union provides aligned signals from both authoritative (articles) and user-generated (tweets) sources, constituting the first resource of this kind for sd. in this paper, we make the following contributions: sd in a controlled domain, such as mergers, could allow model builders to develop deeper insights into the factors influencing model performance. ## background the task. sd is the task of automatically identifying the opinion expressed in a text with respect to a target @xcite . note that sd constitutes a related, but different task than both sentiment analysis and textual entailment. the first considers the emotions conveyed in a text @xcite , while in the second, the goal is to predict whether a logical implication exists between two sentences @xcite . consider the following example: twitter sd. traditionally, research on sd focused on user-generated data, such as blogs and commenting sections on websites @xcite , apps @xcite , and facebook posts @xcite ; above all, mainly due to the handiness of its api, twitter was used as a data source @xcite @xcite @xcite . news sd. at the time of writing, a very small number of sd datasets collecting news have been released, usually building on platforms originally developed by professional journalists, like emergent @xcite or snopes @xcite . note that in twitter sd, the task consists of defining the stance of a tweet with respect to a short target (usually a named entity like hillary clinton @xcite , or a concept like feminism @xcite ); in news sd, on the contrary, the input article is much longer than a tweet, and the target is a complete sentence @xcite . comparison with corpora for news sd. emer-gent @xcite constitutes the first released corpus for news sd: it collects 300 targets and 2,595 articles (with an average of 8.6 articles/target), labeled using a 3-class classification schema. for the first edition of the fake news challenge @xcite , it was enriched with randomly generated unrelated samples. neither of the two corpora is annotated with evidences. to our knowledge, the only other news dataset to be annotated for both sd and er is that of @xcite , which annotates factchecking instances from the debunking website snopes 2 . our work differs in a number of aspects: • statistics. while snopes is larger in size, it provides relatively few samples per target (14,296 samples and 6,422 targets, with an average of 2.22 articles/target); stander, in contrast, collects 3,291 articles on 4 targets, with an average of more than 800 articles/target. ## building the dataset in this section, we report on data collection and annotation, and provide a detailed analysis of the findings from the annotation process. ## baselines and discussion this section provides results for a number of recent techniques. while more complex models could possibly achieve better results, our aim was to set baselines for our dataset with a number of strong models. detailed description of the experimental setting is provided in appendix b and c for replication. ## integrating news and twitter signal as outlined in the introduction, stander contains the same targets as the twitter sd wt-wt corpus @xcite . the union of both corpora thus provides a great opportunity for studying the interplay between authoritative and usergenerated signals: the first refers to long and articulated texts written by professional journalists, while the second refers to a very abundant but potentially noisy stream of posts, which are published without any editorial review. while a detailed time series analysis (lim and tucker, 2019) is beyond the scope of this paper, we provide a first data description and a correlation analysis, which show the potential of the obtained aligned corpus and the challenges it may pose to future research. ## conclusions and future work we presented stander, a new expert-annotated resource for news sd and er. we provided a detailed description of the annotation process and corpus statistics, as well as of the findings from the annotation process. our experiments with a set of strong models indicated a consistent (up to 30%) performance gap between soa and human upper bound: this proves that our corpus constitutes a strong challenge and leaves plenty of room for future work on news sd, er, domain adaptation and multi-task training. moreover, our corpus enables future research in a number of new areas, including: fine-grained er for news sd -where the goal is not only to retrieve evidence snippets, but also their exact location in the text -which goes in the direction of improving interpretability of a model's predictions; and multigenre sd -due to the fact that our corpus aligns with an existing resource for twitter sd -which would open new interesting scenarios in the wider field of rumour verification.