Stance Detection in G erman News Articles (2021)

Abstract: The widespread use of the Internet and the rapid dissemination of information poses the challenge of identifying the veracity of its content. Stance detection, which is the task of predicting the position of a text in regard to a specific target (e.g. claim or debate question), has been used to determine the veracity of information in tasks such as rumor classification and fake news detection. While most of the work and available datasets for stance detection address short texts snippets extracted from textual dialogues, social media platforms, or news headlines with a strong focus on the English language, there is a lack of resources targeting long texts in other languages. Our contribution in this paper is twofold. First, we present a German dataset of debate questions and news articles that is manually annotated for stance and emotion detection. Second, we leverage the dataset to tackle the supervised task of classifying the stance of a news article with regards to a debate question and provide baseline models as a reference for future work on stance detection in German news articles.

## introduction the automatic extraction and analysis of information from text is an important area in natural language processing that tackles a wide range of research problems such as sentiment analysis, veracity and rumor detection, emotion recognition, or fake news detection. with the rapid increase in online content and spread of misinformation, these research topics have gained attention, as it becomes crucial to identify the veracity of the information. stance detection is the task of predicting the position of a text (e.g. in favor or against) towards a specific target, which has shown to aid at identifying the veracity of rumors (rumor classification) and fake news @xcite . as most of * equal contribution the stance detection benchmarks focus on short text snippets, especially tweets, and/or english @xcite , there is a lack of annotated data to address stance detection in longer texts and in other languages. in this paper, we present the cheese dataset, a collection of manually annotated swiss (ch) news articles and debate questions in german for stance and emotion detection. specifically, the cheese dataset comprises 91 debate questions that are automatically paired with related news articles, resulting in a total of 3, 693 question and article pairs. a team of native german speakers manually annotated those pairs with the stance of the article towards the debate question, the article emotion, and the emotion of each individual paragraph. to the best of our knowledge, this is the first available dataset of german news articles annotated for this purpose. while the fake news challenge foot_0 provides document-level stance annotations for english news articles @xcite , other datasets that consider the german language focus on shorter texts @xcite . finally, we perform target-specific stance detection as a supervised task using our annotated data. we then build baseline models as a reference in future work on stance detection in german news articles. since the dataset provides both stance and emotion annotations, future work could assess the use of emotion information to improve stance detection in a multi-task setting (see discussion in section 5). moreover, computational journalism applications, such as news recommender systems, could leverage the dataset to mitigate the selective exposure of one-sided articles @xcite . the cheese dataset and the code to reproduce the baseline models are publicly available. 2 ## related work most of the available annotated datasets for stance detection focus on short texts such as tweets. the semeval-2016 shared task hosts a dataset of annotated english tweets and their stance towards the targets 'atheism', 'climate change', 'feminism', 'hillary clinton', and 'abortion' @xcite . aside from english, there are available tweets datasets for catalan and spanish @xcite , french @xcite , or italian @xcite . for german, the multilingual dataset x-stance offers a collection of political questions and their corresponding comments written by candidates of elections, partly in german, partly in french and italian @xcite . in the journalistic domain, there are two available datasets of english news articles. the emergent dataset is a collection of claims and their associated news articles @xcite . similarly, the fake news challange detection task (fnc-1) offers a dataset of news articles and their stance towards a headline @xcite . while the emergent dataset performs the stance detection task on single-sentence headlines towards a specific claim, the fnc-1 is a documentlevel stance detection task that considers the entire news article. in our dataset, we also address stance detection at document level, but in german text. emotion detection datasets are also mostly concerned with short text snippets, such as tweets @xcite , news headlines @xcite , textual dialogues @xcite , or sentences in fairy tales @xcite . in contrast, our dataset offers articleand paragraph-level annotations for emotion detection in german news articles. paragraph-level annotations could be leveraged to identify and analyse patterns of emotion sequences (i.e. emotion flow) in news articles. while emotion flow has been explored in books @xcite and movie synopsis @xcite , such analysis has not yet been applied to journalistic content. to improve stance detection, some research approaches focus on using sentiment information, and more specifically, the polarity of the text (i.e. positive, negative, or neutral). in fact, the annual workshop on semantic evaluation (semeval) organised a stance detection shared task and provided a dataset of tweets also annotated with sentiment information @xcite . 3 however, the conclusions on whether sentiment information aids at improving stance detection are mixed. some studies report the benefits of using such information @xcite , whereas others claim that the tasks are independent of each other. for example, a text with a positive sentiment could show an opposing stance towards a specific claim @xcite @xcite . ## the cheese dataset the cheese dataset is a collection of 3, 693 pairs of debate questions and swiss news articles in german that are manually annotated for stance and emotion detection. specifically, the dataset comprises a total of 1, 970 different articles paired with up to six questions (∼1.87 on average) from a list of 91 questions that we manually compiled to ensure a variety of debate topics (section 3.1). the data comes from ∼670k news articles provided by the swiss media companies blick (∼536k articles), 4 neue zürcher zeitung (∼125k articles), foot_4 and nzz am sonntag (∼9k articles). 6 the articles ## stance detection task in the stance setting, the cheese dataset provides with (a) a natural-language question and its set of related keywords; (b) a news article; (c) the stance of the article towards the question. similar to the stance detection task defined in the fnc-1, we use the stance classes ja, dafür ('for'), nein, dagegen ('against'), diskutierend ('discussion'), and kein bezug ('unrelated'). the target-specific stance detection task is then a multiclass classification task that consists of predicting an article's position towards its associated question. we assess different baselines on the task using the cheese dataset as a reference for future work. the proposed baselines are a bag-of-words linear classifier and a bert-based model @xcite ., 14 https://huggingface.co/ bert-base-multilingual-uncased label two-pass two-pass-ext all diskutierend 620 154 774 ja, dafür 509 193 702 nein, dagegen 232 54 286 kein bezug 403 1,025 1,428 unklar 332 171 503 since the amount of training data is relatively small compared to other similar datasets, such as xstance or fnc-1 (see ## discussion and future work the emotion of a text is closely related to sentiment analysis. zhang et al. (2020), for example, leverage semantic-emotion knowledge in a cross-target stance detection task with positive results. however, there is insufficient research on combining emotion and stance information; perhaps due to the lack of datasets with both annotations. since the cheese dataset is annotated for both stance and emotion detection tasks, it allows the research community to further experiment in this direction. in our experiments, we fine-tune a german pretrained bert model on our dataset. bert-like models are able to deal with input sequences of up to 512 tokens. since our dataset consists of news articles that are on average 843 and up to 4, 130 tokens long, it would be interesting to assess the impact on stance detection when considering longer texts. therefore, in future work we plan to experiment with model architectures that handle long input sequences, such as the longformer, which processes sequences up to 4, 096 tokens long and even up to 16k with the current gpus @xcite . the size of the cheese dataset is considerably smaller than other stance datasets as shown in we also expect the dataset to be a valuable resource in computational journalism, for example, to generate diverse news article portfolios or to provide a balanced exposure of articles' positions with recommender systems. ## conclusion we presented the cheese dataset, a manually annotated dataset of swiss news articles. in contrast to most of the available datasets, which focus on short texts and/or the english language, the novelty of our dataset is that it provides document-level annotations on german news articles. specifically, the dataset consists of a collection of german news articles automatically paired with debate questions. a team of native german speakers labelled each article and question pair with the stance of the article towards the question, the article emotion, and the emotion of each article's paragraph. we then performed target-specific stance detection as a supervised task using the cheese dataset and provide baseline models as a reference in future research on stance detection in german news articles. empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 833-844, hong kong, china. association for computational linguistics. andrew yates, rodrigo nogueira, and jimmy lin. 2021. pretrained transformers for text ranking: bert and beyond. in proceedings of the 2021 acm international conference on web search and data mining (wsdm), page 1154-1156, online. association for computing machinery. bowen zhang, min yang, xutao li, yunming ye, xiaofei xu, and kuai dai. 2020. enhancing crosstarget stance detection with transferable semanticemotion knowledge. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 3188-3197, online. association for computational linguistics. elena zotova, rodrigo agerri, manuel nuñez, and german rigau. 2020. multilingual stance detection in tweets: the catalonia independence corpus. in proceedings of the 12th language resources and evaluation conference, pages 1368-1375, marseille, france. european language resources association.