TATA : Stance Detection via Topic-Agnostic and Topic-Aware Embeddings (2023)

Abstract: Stance detection is important for understanding different attitudes and beliefs on the Internet. However, given that a passage’s stance toward a given topic is often highly dependent on that topic, building a stance detection model that generalizes to unseen topics is difficult. In this work, we propose using contrastive learning as well as an unlabeled dataset of news articles that cover a variety of different topics to train topic-agnostic/TAG and topic-aware/TAW embeddings for use in downstream stance detection. Combining these embeddings in our full TATA model, we achieve state-of-the-art performance across several public stance detection datasets (0.771 F_1 -score on the Zero-shot VAST dataset). We release our code and data at https://github.com/hanshanley/tata.

## introduction stance detection is the task of determining the attitude of a piece of text with regard to a particular topic or target. stance detection is often utilized to gauge public opinion toward products, political issues, or public figures. for example, the statement "i think abraham lincoln would make a good president" would have a pro stance concerning the topic of abraham lincoln. simultaneously, the same text would be neutral to the topic of fruits. several works have built topic-specific stance classifiers that have achieved promising results for particular targets @xcite ; however, these same approaches have often failed to generalize to topics outside of their training domain, limiting their real-world use. recent works have sought to alleviate this issue by utilizing general stance features @xcite , common sense reasoning @xcite , and generalized topic representations @xcite @xcite , achieving better performance in predicting stance on previously unseen topics (zero-shot stance detection) and on topics for which there are few training examples (few-shot stance detection). in this work, building on these insights, we create and release synthetic datasets for designing topic-aware/taw embeddings and generalized topic-agnostic/tag stance embeddings for use in downstream stance classification tasks. taking advantage of both the topic-aware and topic-agnostic perspectives and embeddings, our full tata model achieves stateof-the-art performance in both the zero-shot and few-shot stance classification settings on the vast benchmark dataset @xcite . specifically, utilizing an unstructured news article dataset (hanley and durumeric, 2023) and extracting topics from within particular passages using the t5-flan language model @xcite , we train a topic-aware/taw embedding layer with a triplet loss @xcite . by using triplet loss, we force training elements with similar topics to have similar embeddings and training elements with different topics to have dissimilar embeddings. by using an unlabelled dataset of news articles with a diverse and wide set of automatically extracted topics when creating these topic embeddings, rather than simply relying on the set of topics with expensive and carefully labeled stance datasets as in other works @xcite , enables these topic embeddings to incorporate relationships between many different topics. similarly, for our topic-agnostic/tag embeddings, after first extending the original vast training dataset with paraphrased versions of the original texts, we utilize contrastive learning objectives to then extract generalized stance features @xcite . by using contrastive learning, we force training instances with the same stance to have similar embeddings while simultaneously forcing training examples with different stances to have different embeddings. this enables our tag layer to model general stance features that are important for classifying the stance in a given example. combining both the topic-aware/taw and topicagnostic/tag perspectives into one model tata, we achieve state-of-the-art performance on the vast dataset in both zero-shot (0.771 f 1 -score compared to 0.723 in prior work @xcite ) and few-shot (0.741 f 1 -score compared to 0.715 in prior work) settings. our tata model further achieves competitive results on the sem16t6 dataset @xcite in the zero-shot setting. by training on synthetic datasets, taking advantage of stance features that are common across different topics/tag, and deriving specific features particular to given topics utilizing a rich embedding latent/taw, our tata model thus achieves stateof-the-art performance on current benchmarks for stance classification. given its real-world applications, several works have sought to improve upon domain-specific and baseline stance-detection methods to identify stances in few-shot settings (where the classifier is evaluated on a large number of topics for which it had few training examples) and zero-shot scenarios (where the classifier is evaluated on new, previously seen topics) @xcite @xcite . for example, in addition to developing one of the most varied benchmark datasets for evaluating zero-shot stance detection vast, @xcite also pioneered the use of generalized topic representations to perform zero-shot and few-shot stance detection. more recently, @xcite , improving upon prior work, utilized targeted outside knowledge to add information for stance detection on previously unseen topics. @xcite , utilized generative models (e.g., gpt2) to create synthetic data to engineer training data for unseen topics. finally, @xcite further proposed target-aware graph contrastive learning to learn relationships between topic-based representations for downstream stance classification. constrastive learning. contrastive learning, which seeks to build representations by differentiating similarly and differently labeled inputs, has become an increasingly popular version of selfsupervised learning @xcite @xcite . contrastive learning is often utilized to build robust representations that can then be utilized in downstream tasks @xcite including generating sentence embeddings @xcite , image classification @xcite , and image captioning @xcite . as in our work, contrastive learning has also increasingly been utilized to model stance features @xcite . for example, @xcite utilized in-batch contrastive learning to differentiate between examples of stance classes while also building a graph of prototypical topics to train topicspecific features for downstream stance detection. ## topic-aware/taw dataset as previously stated, within this work, we seek to build informative topic-aware/taw embeddings that can then be utilized in downstream stance detection. as such to pre-train these embeddings, we collect a new dataset that includes a wide range of topics. this new dataset, designed for creating topic-aware embeddings, consists of sets of quadruplets such that d t aw ## augmented vast dataset in addition to developing our own dataset for training taw embeddings, we further extend allaway and mckeown (2020)'s vast dataset to pre-train our topic-agnostic/tag embeddings. as previously noted, the vast dataset is one of the most varied stance datasets. vast consists of a dataset to create paraphrases of the original vast dataset's passages, we rely on the pre-trained dipper paraphraser foot_3 , a t5-based model fine-tuned to paraphrase paragraph-level texts @xcite . see appendix c for details on the hyperparameters we utilized for dipper and see appendix e for example dipper paraphrases. by utilizing the t5-based dipper model, we augment the original vast dataset without changing the original meaning of the original vast passages. as with our taw dataset, we further augment vast with different paraphrases of the original topic strings with the same methodology outlined in section 3.2. by paraphrasing these topics, we again model the different ways of expressing the same topic. altogether, by paraphrasing each row from the vast dataset (paraphrasing individual texts as many as 16 times) and by extracting out as many as 10 paraphrases of each topic phrase/word within the original vast dataset d v ast , we extend the original training dataset of 13,477 examples to a total of 743,644 different examples d v astaug . ## methods we develop a model (tata) that combines topicaware and topic-agnostic embedding layers to perform zero and few-shot stance detection. this model (figure 1 ) consists of the topic-aware embeddings layer (5.1), the topic-agnostic embedding layer (5.2), two attention layers using the output of the topic-aware embedding layer and the topicagnostic embedding layer (5.3), and finally a twolayer feed-forward neural network for stance classification (5.4). ## experimental results our models achieve a noticeable increase in performance over prior work. we present our results on the vast test dataset in ## conclusion combining pre-trained layers that learn topicagnostic/tag features and topic-aware/taw features, our tata model achieves state-of-the-art performance on the vast test dataset. tata performs better than prior works not only in few-shot settings but also in zero-shot settings. our most common failure mode is cases where a given passage has both positive and negative stances within it. for future work, during training, we plan to incorporate additional passages with multiple stances toward different topics.