Is Stance Detection Topic-Independent and Cross-topic Generalizable? - A Reproduction Study (2021)

Abstract: Cross-topic stance detection is the task to automatically detect stances (pro, against, or neutral) on unseen topics. We successfully reproduce state-of-the-art cross-topic stance detection work (Reimers et. al, 2019), and systematically analyze its reproducibility. Our attention then turns to the cross-topic aspect of this work, and the specificity of topics in terms of vocabulary and socio-cultural context. We ask: To what extent is stance detection topic-independent and generalizable across topics? We compare the model’s performance on various unseen topics, and find topic (e.g. abortion, cloning), class (e.g. pro, con), and their interaction affecting the model’s performance. We conclude that investigating performance on different topics, and addressing topic-specific vocabulary and context, is a future avenue for cross-topic stance detection. References Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. Classification and Clustering of Arguments with Contextualized Word Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 567–578, Florence, Italy. Association for Computational Linguistics.

## topic specifics in classification to support the conclusions in @xcite on the success of cross-topic stance detection, we expect a relative stability of performance over topics. the following sections go into some details not explored in @xcite , specifically the cross-topic performance of different topics, and the interaction between topic and class and its influence on performance. ## conclusion: topic matters stance detection is a difficult nlp task. despite recent advances by pre-trained transformers, these models have similar issues in a cross-topic setting as earlier models. this paper reproduced stance detection experiments with pre-trained transformers by @xcite , training on seven topics and testing on an eighth topic. we found similar results, but also both class and topic influencing performance. cross-topic bert models perform below mean model performance in some topics (abortion, gun control) on the pro and con classes. this makes us pause about reimers et al. secondly, we also think it is necessary to analyze the context of topics, and its relation to other debate topics within and outside the dataset. most topics in stance detection studies are currently u.s. socio-political issues. this goes beyond a limitation of language, such as a focus on english without specifying this @xcite , since the same socio-cultural topics are not even universally relevant in the english-speaking world (gun control is not a salient discussion in scotland). such a focus on topic diversity is also important for usecases. for diversity of viewpoints in search @xcite or news recommendation @xcite , stance detection needs to work on many different topics. @xcite states that we need to carefully define specific nlp tasks and capabilities needed to solve them. modelling cross-topic stance detection in a topic-agnostic manner, while divorcing it from socio-cultural context, might not do justice to stance detection. future work might focus on the specifics of topics: analyzing similarity between discussions @xcite , or modelling required socio-cultural contextual knowledge ('second amendment is related to gun control'). models able to deal with topic-specific vocabulary and socio-cultural context of debates might improve on the state-of-the-art of cross-topic stance detection.