Synthetic Examples Improve Cross-Target Generalization: A Study on Stance Detection on a T witter corpus. (2021)

Abstract: Cross-target generalization is a known problem in stance detection (SD), where systems tend to perform poorly when exposed to targets unseen during training. Given that data annotation is expensive and time-consuming, finding ways to leverage abundant unlabeled in-domain data can offer great benefits. In this paper, we apply a weakly supervised framework to enhance cross-target generalization through synthetically annotated data. We focus on Twitter SD and show experimentally that integrating synthetic data is helpful for cross-target generalization, leading to significant improvements in performance, with gains in F1 scores ranging from +3.4 to +5.1.

## introduction stance detection (sd) is a widely investigated task @xcite , which constitutes an important component of many complex nlp problems, ranging from fake news detection to rumour verification @xcite @xcite . since from early works @xcite , research on sd focused on user-generated content, ranging from blogs and commenting sections on websites @xcite , to reddit or facebook posts @xcite and, above all, twitter data @xcite . recently, @xcite released will-they-won't-they (wt-wt), a very large corpus of stance-annotated tweets discussing five us mergers and acquisitions (m&a) operations spanning over two industries: healthcare and entertainment. m&a is a general term that refers to the process in which the ownership of companies are transferred. such process has many stages that range from informal talks to the closing of a deal, and discussions may not be publicly disclosed until a formal agreement is signed @xcite : in this sense, the analysis of the evolution of opinions and concerns expressed by users about a possible m&a operation, from early stage discussion to the signing of the merger (or its rejection), is a process similar to rumor verification, a widely studied field @xcite . interestingly, @xcite observed a consistent drop in performance when a system trained on mergers in one industry is tested on data discussing a merger in a different industry. such a performance drop when testing conditions deviate from training conditions is a known problem in stance detection (sd) @xcite . in this paper, we investigate the impact of using synthetically annotated data to improve zero-shot cross-target generalization in twitter sd: (1) we investigate a weakly supervised framework for sd, which integrates synthetically annotated data to improve performance on new targets; as to our knowledge, we are the first to use synthetically annotated data for sd; ## experimental setup data. we consider the following data ( ## experiments and discussion baseline. experiment i. to understand the impact of including different types of synthetic data during training, we consider three settings: for experiments, we randomly add synthetic samples with a proportion of 50% w.r.t. the train set size; to account for uncertainty, we use sample weighting for synthetic samples: sup, ref and com are weighted 0.6, while unr are weighted 0.2 (after qualitative analysis, we found them to be noisier). results in experiment ii. we consider the best performing setting, related mergers, and perform a second set of experiments to understand the impact of adding synthetic samples belonging to different stances; we consider: only unr; only com; only unr+com; sup+ref +com; sup+ref ; and finally adding samples from all stances. differences in performance between settings are negligible ( ## conclusions and future work we investigated an inexpensive framework to integrate unlabeled id data to improve cross-target sd. we studied twitter sd and showed, through a comprehensive set of experiments, that it is a promising strategy. we reserve to study its applicability to other domains in future work.