Exploration of Contrastive Learning Strategies toward more Robust Stance Detection (2023)

Abstract: Stance Detection is the task of identifying the position of an author of a text towards an issue or a target. Previous studies on Stance Detection indicate that the existing systems are non-robust to the variations and errors in input sentences. Our proposed methodology uses Contrastive Learning to learn sentence representations by bringing semantically similar sentences and sentences implying the same stance closer to each other in the embedding space. We compare our approach to a pretrained transformer model directly finetuned with the stance datasets. We use char-level and word-level adversarial perturbation attacks to measure the resilience of the models and we show that our approach achieves better performances and is more robust to the different adversarial perturbations introduced to the test data. The results indicate that our approach performs better on small-sized and class-imbalanced stance datasets.

## introduction a controversial topic divides people into two groups with different views (support/against) on the topic of discussion. some popular, controversial topics include the legalization of abortion, concern about climate change, gay marriage, obama, the legalization of marijuana, feminism, and atheism. the existing stance detection models are non-robust, and even simple perturbations in the input sentences affect the model's performance @xcite . for example, the input sentence 'fetus is not human' has the stance label of 'support' for the topic of 'legalization of abortion.' however, when there is a variation to the same input sentence, such as 'a bunch of cells is not human,' it will confuse the model in reproducing the same stance label of 'support.' also, spelling errors, missing words, repetition of words, and other commonly occurring errors in the text are the adversarial errors that make the stance detection models fall short in detecting the stance compared to humans. we aim to make the stance detection system more robust to adversarial perturbations by accommodating the variations and errors in the text when detecting the stance. we primarily concentrate on binary stances (e.g., support/against) in social media for english texts, such as tweets, news comments, and discussion forums. we use the contrastive learning (cl) approach to construct more robust sentence representations for the stance detection task. given an example we call anchor, the cl technique brings the similar example closer to the anchor and drives the dissimilar example away from the anchor in the representation space. we build similar (positive) and dissimilar (negative) examples for cl by considering the stance label of the examples. we mainly explored different strategies for building positive and negative examples for an anchor example to learn the sentence representations in a contrastive fashion. along with cl, we use masked language modeling as a token-level objective to learn textual representations (see figure 1 ). our code is available in the github repository foot_0 . we make the following contributions. ## related work many approaches @xcite @xcite @xcite @xcite @xcite were proposed to tackle different problems in the stance detection task. however, the existing stance detection models are sensitive to adversarial errors, and changes in the vocabulary of the input sentences @xcite . the adversarial robustness of the model is measured by making the model predict against the test set with char-level, and sequence-level modifications to the input as well as with the word substitutions @xcite @xcite . @xcite used various perturbations for char-level such as insertion, deletion, replacement, etc., and word-level perturbations such as replacement with synonyms, negation, etc. @xcite used the resilience score introduced by @xcite to measure the robustness of the model. cl is used to acquire better representations of text for many natural language tasks such as question-answering @xcite , multiple choice video questions, text-to-video retrieval @xcite ), text summarization @xcite @xcite etc. @xcite used contrastive learning to learn noise invariant sentence representation with the help of different sentence-level augmentation strategies like span deletion, substitution, and reordering. @xcite introduced a hierarchical contrastive learning strategy to improve the zeroshot stance detection (zssd) task by capturing the relationships not only between target-invariant and target-specific features but also among various stance labels. in this study, our objective is to develop and explore a range of strategies encompassed within contrastive learning. our aim is to enhance the quality of document representations specifically for the task of stance detection, consequently bolstering the robustness of stance detection classification models. training examples fetus is not human triplet mining strategy random / hard / hard & easy .. . fetus is not human distilroberta encoder contrastive learning mlm + contrastive loss mlm loss . . . figure 1: architecture diagram for learning sentence representations using cl and mlm objectives to further use in the stance detection task. ## experiments we have chosen seven stance detection datasets, debateforum (df) (hasan and ng, 2013), se-meval2016 (se) @xcite , arc @xcite , perspectrum @xcite , fnc-1 (pomerleau and rao), ksd-biden and ksd-trump @xcite for the experiments. we have retained only the examples that have support/against equivalent labels in the datasets as we mainly focus on binary stances. out of the seven chosen datasets, the perspectrum dataset has more instances (11825), ksd-biden has the least number of instances (766) and fnc-1 is the most imbalanced dataset (78/22). see ## conclusion in this work, we have adopted the combination of cl + mlm method and explored different triplet strategies to learn more robust sentence representations to use in the stance detection task. experiment results show that our proposed methodology is more resilient to errors and variations. also, the experiments with different setups show that our proposed methodology is effective for small-sized as well as class-imbalanced datasets.