Adversarial Training for News Stance Detection: Leveraging Signals from a Multi-Genre Corpus. (2021)

Abstract: Cross-target generalization constitutes an important issue for news Stance Detection (SD). In this short paper, we investigate adversarial cross-genre SD, where knowledge from annotated user-generated data is leveraged to improve news SD on targets unseen during training. We implement a BERT-based adversarial network and show experimental performance improvements over a set of strong baselines. Given the abundance of user-generated data, which are considerably less expensive to retrieve and annotate than news articles, this constitutes a promising research direction.

## introduction stance detection (sd) is an important nlp task @xcite with widespread applications, ranging from rumor verification @xcite and fact checking @xcite . traditionally, research in sd focused on user-generated data, such as twitter or reddit @xcite : this is mainly due to the abundance of such data, which are usually freely available online; moreover, user-generated data tend to be relatively short and compact, and thus more affordable to annotate and process. starting from popular shared tasks such as @xcite , sd on complex and articulated input, such as news articles, has gained increasing popularity. notably, effective news sd would constitute an invaluable tool to enhance the performance of human journalists in rumor and fake news debunking @xcite . in line with the general trend in nlp, deep learning-based models have long since established state-of-the-art results in news sd @xcite . notably, training neural networks relies heavily on the availability of large labeled datasets, which are especially expensive to obtain for items such as news articles. as a consequence, following research on other text classification tasks such as sentiment analysis @xcite , research in sd investigated effective methods for cross-domain sd, where the scarcity of data for a specific dataset is supplemented with stance-annotated data from other domains. in this context, preliminary research in adversarial domain adaptation obtained promising results for both twitter @xcite and news @xcite sd. in this paper, we focus on the new task of crossgenre sd: we consider adversarial knowledge transfer from two datasets, wt-wt and stander, which collect samples in the same domain (i.e. the financial domain), but which belong to different genres (i.e. twitter and news). we show experimentally that improvements in news sd performance can be achieved through cross-genre sd, which constitutes a promising direction for future research. ## an aligned multi-genre stance detection corpus in this work, we rely on two recently released datasets for news and twitter sd: the stander corpus for the news genre @xcite , and the wt-wt corpus for twitter @xcite . both corpora collect samples discussing four mergers and acquisition (m&a) operations in the healthcare industry ( samples in both stander and wt-wt are manually stance-labeled by domain experts using a four-classes annotation schema distinguishing between support, refute, comment and unrelated, which expresses the sample's orientation about the outcome of the m&a (succeeded or rejected). as observed in @xcite , the two corpora present comparable signals, but display different characteristics which reflect the diverse genres they belong to. the twitter samples are abundant and noisy, as indicated by the high percentage of unrelated and commenting samples (figure 1 and ## experimental setting baselines. we report results with the three baseline models proposed in @xcite : a dummy random and majority vote baseline, abd bertemb, an mlp leveraging sentence-bert embeddings (reimers and gurevych, 2019); moreover, we also consider two further baselines: training setting and preprocessing. we train in a cross-target setting (train on three mergers, test on the fourth) with the adam optimizer. for each configuration, we randomly select 20% of the training samples as heldout data. for experiments with adversarial cross-genre sd, we randomly select a number of twitter samples equal to the news training samples (i.e. we double the size of the training set). this ratio was found to perform best in preliminary experiments (refer to the appendix for further details). the test set contains news samples only. we lowercase both tweets and news samples. hyperparameters. we set 128 as maximum sample length (including special tokens). we initialize our architecture with bert large uncased foot_0 . bert's weights are updated during training. we train using adam (kingma and ba, 2015) on baatches of 23 samples, for a maximum of 70 epochs, with early stopping monitoring the sd loss on the development set. evaluation. as in @xcite , and in line with other works on news sd @xcite , we report on macro-averaged f 1 and consider both per-target operation scores, and average scores weighted by target operation size. for the adversarial cross-genre experiments, we compute accuracy for the binary gi task. for computing the evaluation metrics, we use the sklearn's implementations foot_1 . ## experiments and discussion in this section, we report on our cross-genre sd experiments. the discussion is organized around three research questions. rq1 what is the effect on news sd of including annotated data from a different genre? as shown in rq2 is adversarial training effective to improve cross-genre sd? adding an adversarial component to the bert cot rain model leads to gains in performance over all considered targets, with improvements ranging from +0.5 (aet hum) and +4.0 (antm ci) in macro-averaged f 1 score. ( ## conclusions in this paper, we discussed the new task of crossgenre sd: our experiments with a range of bertbased architectures show that partially obfuscating the genre component through adversarial training leads to better generalization, especially considering low-frequency labels. cross-genre sd thus constitutes a promising future research direction. future work might include experiments using different underlying feature extractors, such as roberta, or with adapters, to study the robustness of cross-genre sd over modeling choices. the integration of cross-genre and cross-domain adaptation, possibly in a multi-task setting as in @xcite , also offers interesting ideas for future investigation.