SSSD : Leveraging Pre-trained Models and Semantic Search for Semi-supervised Stance Detection (2023)

Abstract: Pre-trained models (PTMs) based on the Transformers architecture are trained on massive amounts of data and can capture nuances and complexities in linguistic expressions, making them a powerful tool for many natural language processing tasks. In this paper, we present SSSD (Semantic Similarity Stance Detection), a semi-supervised method for stance detection on Twitter that automatically labels a large, domain-related corpus for training a stance classification model. The method assumes as input a domain set of tweets about a given target and a labeled query set of tweets of representative arguments related to the stances. It scales the automatic labeling of a large number of tweets, and improves classification accuracy by leveraging the power of PTMs and semantic search to capture context and meaning. We largely outperformed all baselines in experiments using the Semeval benchmark.

## introduction stance detection (sd) is the task that automatically determines whether the author of a text is in favor of, against or does not manifest about a given target. targets can be companies, movements, people or ideas @xcite . it was initially applied to the analysis of political debates in online forums and has become very attractive to measure public opinion on social networks @xcite . sd on social media can be categorized based on different criteria, including the type of target, the type of stance (i.e., in favor, against, or neutral), and the level of analysis (i.e., post level or network level). the features used for classification vary according to the analysis level: textual features only (post level) or user-related attributes and behaviors such as mentions and the number of followers (network level) to improve the model accuracy @xcite . the state-of-the-art methods for sd @xcite are based on machine learning (ml) and have shown to be effective in various scenarios @xcite . however, they rely on manual and complex feature engineering, particularly when applied at the network level. on the other hand, deep learning (dl) based methods for sd @xcite do not require feature engineering, but they can easily overfit if not trained with enough labeled data, due to their high number of parameters @xcite . unfortunately, labeling data is an expensive and time-consuming task, leading to small labeled datasets for specific domains @xcite . transfer learning @xcite and unsupervised approaches @xcite @xcite are promising directions for sd, but they still face challenges in achieving comparable results to supervised machine learning approaches, especially in highly polarized environments such as twitter. this is due to the difficulty of detecting stances in a noisy and polarized platform such as twitter, where people express their opinions in nuanced and complex ways. despite these challenges, researchers continue to explore new approaches to improve the accuracy of sd in various contexts @xcite . using pre-trained models (ptms) based on the transformers architecture @xcite , such as bert @xcite and gpt @xcite , researchers can address the challenge of data scarcity and the variability and noise inherent in twitter, while capturing the relevant semantic and contextual information needed to classify stances accurately. ptms are trained on massive amounts of data and can capture nuances and complexities in linguistic expressions, making them a powerful tool for detecting stances. by finetuning these models on smaller labeled datasets, they can learn specific patterns of stances in different contexts, which can help overcome the challenges of variability and noise on twitter. additionally, ptms can be used to search or compare tweets with similar stances through cosine similarity @xcite , aiding the task of stance detection. ptms represent a promising approach to improving the accuracy of ds on twitter. in this paper, we propose sssd (semantic similarity sd), a semi-supervised method for stance detection on twitter that leverages ptms and semantic search to automatically label a large, domainrelated corpus for training a stance classification model. the method assumes as input a domain set of tweets about a given target and a labeled query set of tweets of representative arguments related to the stances. the tweets of the domain and query sets are converted into a contextual representation using a ptm, such that a similarity function can identify the semantic proximity of the tweets of both sets. for each tweet of the query-set, the search function selects the k most similar tweets from the domain set, assigning them the respective stance label. this set of labeled tweets is then used to train an sd classification model using some ml classification algorithm. the remaining unlabeled tweets can be classified using this model. ssds improves stance classification performance by leveraging the power of ptms and semantic search to capture the context and meaning of tweets in a specific domain, addressing the complexity of stance labeling. it reduces the need for manual annotation, an expensive and time-consuming task, enabling the accurate automatic label of a large volume of tweets with minimal computational costs. our experimental setting involved three classification algorithms and the sd benchmark datasets and metrics @xcite , which includes six targets. sssd outperformed the baselines (al-ghadir et al., 2021) by 13.9 percentage points (pp) and @xcite by 11.2 pp in the overall averaged f-measure metric. we also assessed the influence of the value of k on the similarity of retrieved tweets, number of labeled tweets and stance classification performance. the main contributions of our study can be summarized as follows: the remaining of this work is structured as follows. section 2 presents the related work. section 3 details the proposed semi-supervised sd method. section 4 describes the experiments. section 5 outlines conclusions and future work. ## related work stance detection is a complex form of subjectivity analysis that focuses on identifying the attitude or perspective that a speaker or writer has towards a particular topic or issue. unlike sentiment analysis (i.e., positive, negative), sd attempts to identify more subtle variations in the speaker's position, such as whether they are in favor of or against a particular policy or support or oppose a particular political candidate @xcite . the task of sd gained significant popularity following the launch of a competition on twitter during semeval 2016. two tasks were proposed: supervised approaches (task a) and unsupervised/semisupervised approaches (task b). the competition provided labeled data encompassing different targets and a well-defined methodology to assess the solutions, with a common evaluation metric @xcite . most studies in sd for the english language rely on semeval datasets and evaluation methodology as a benchmark, which are limited in scope and size. the semeval datasets cover only a specific set of domains and targets, and their small size may not capture the full complexity of the task, leading to overfitting or generalization issues. therefore, it is important to create new datasets that can expand the scope of research in stance detection to other domains, languages, and targets (aldayel and magdy, 2021). as a reflection of the scarcity of labeled data, state-of-the-art sd methods heavily rely on complex feature engineering techniques, making their reproduction a challenging tasks. for example, the leading sd system @xcite utilizes sentiment lexical dictionaries and ranked lists of tf-idf weighted words to train k-nn classi-fiers, but its operational details are unclear, hindering its reproducibility @xcite . other studies @xcite @xcite leverage network information (e.g., hashtags, retweets) to enhance classifier performance. however, these approaches require additional user behavior data, which limits their applicability beyond social media platforms. recent studies have focused on developing unsupervised sd models using clustering techniques. the system in @xcite ) used clustering at the author and topic levels, leveraging both the content and interaction networks of the users. clustering was leveraged in @xcite to create an initial set of stance partitions for annotation and showed that retweets as a feature provided the best performance score upon implementing the clustering algorithm. the work in @xcite introduced embedding representations of users' tweets to enhance the sd model using hierarchical clustering to analyze fine-grained polarization between groups of tweets related to the turkish election. while unsupervised methods are useful for minimizing the need for manual labeling, they generally perform worse than supervised methods when labeled data is available. some unsupervised approaches @xcite still require some level of human supervision or adjustment, but this can be done more quickly than the manual labeling of large datasets. to address the limited availability of labeled data for sd tasks, some studies @xcite have incorporated transfer learning techniques. these works involve fine-tuning a pre-trained language model on the source target data to learn a target-specific semantic-emotion representation. the resulting representation is then used to train a classifier for stance detection on the target with limited labeled data. by leveraging the transferred representation, which encodes information about the semantic and emotional characteristics of the target, the classifier can be trained with a smaller number of labeled examples @xcite . the transfer learning approaches crossnet and textcnn-e were proposed in @xcite for enhancing sd across multiple targets. however, this approach requires a large labeled dataset and falls short of surpassing current state-of-the-art systems in sd. works as @xcite have proposed transformer-based architectures combined with data augmentation and fine-tuning. they trained specific sentence classifiers based on umberto using auxiliary datasets from tasks like sentiment analysis, irony detection, and hate-speech detection. the resulting labels were then augmented as new sentences in the sardistance dataset. this training dataset was expanded by labeling additional tweets using distant supervision based on specific hashtags. similarly, @xcite utilized wikipedia articles to extract knowledge for each topic in a seven-themed dataset. these studies incorporated the concept of transfer learning by utilizing new datasets beyond the semeval stance task. in summary, complex feature engineering techniques and network information can improve the performance of sd classifiers, but they are difficult to reproduce and not practical for use in contexts other than social media. unsupervised methods can minimize the need for manual labeling but generally perform worse than supervised methods when labeled data is available. transfer learning techniques are useful for addressing the limited availability of labeled data and can be used with smaller labeled examples, but some approaches require a large labeled dataset. we contribute to the field by proposing a novel semi-supervised method that leverages the ptms and semantic search to automatically label a large domain-related corpus and train an accurate stance classification model. this approach reduces the need for manual and costly annotation efforts, enabling labeling a large volume of tweets with minimal computational costs. ## sssd overview sssd is a novel approach to conducting sd on twitter using ptms and semantic search. it explores ptms to capture the semantic and contextual meaning of tweets, taking advantage of the strengths of deep learning-based approaches. by leveraging the power of ptms and semantic search, we aim to automatically label a domain corpus for training sd models. ptms are pre-trained on extensive text data to acquire general language representations that can be further fine-tuned for specific tasks such as sd on twitter. sssd is semi-supervised: it relies on a set of labeled queries as input to the semantic search algorithm that automatically labels a larger corpus of domain-related tweets, which is then used to train a stance classification model. this reduces the effort required to label a large volume of tweets, while still achieving good classification performance. by using semantic search to identify the most relevant tweets for each query, sssd can focus on the most important posts for the stance classification problem while ignoring irrelevant or noisy data. this is an advantage compared to unsupervised approaches, which may struggle to identify the most relevant data, especially in noisy and complex datasets like twitter. the remaining of this section describes the input data required by ssds, and the semantic stance detection process. ## experiments our experiments were designed to assess the performance of sssd against baseline systems and the influence of the value of k in our results. in this section we describe the data and chosen baselines, and detail the experiments. all our experiments are reproducible, and the code and tools used in their development are available in a public repository 1 . ## conclusions in this work, we proposed sssd, a semi-supervised method for sd on twitter based on semantic search. we leverage ptms in combination with a topk function to retrieve and label domain-specific tweets, which are then used the automatic label a dataset to train a supervised classification model. it reduces the dependence on large annotated datasets while significant improving classification performance. we largely outperformed state-of-the-art supervised systems using the semeval stance detection benchmark. in our evaluation, we tested different k values, assessing their impact on performance with various datasets and classifiers. the results showed that our method is robust and has a high degree of generalization. we also found that the optimal k varied based on the specific scenario, with a trade-off between similarity scores and the number of labeled tweets to maximize ranking performance. overall, our findings indicate that our method is effective for various sd scenarios, but the value of k needs to be identified experimentally. we have shown that by leveraging ptm and semantic search, our method handled the nuances and complexities of stance automatic labeling. our approach is simple, computationally inexpensive, and the encouraging results motivates us to further investigate it in other text classification tasks, making it a valuable contribution to the field of nlp by addressing the challenge of labeled data scarcity. as future work, we intend to qualitatively evaluate our method regarding some challenges faced when analyzing social phenomena on twitter. one of them is the bias introduced in the interpretation of topics due to hashtags to represent the objects of study. a common example is false positives, where a tweet is falsely inserted in the context of a hashtag by refuting the idea represented by it, usually through replies. there is also the scenario where a hashtag is purposefully linked to events (e.g. games, famous artists) outside of its context to increase its relevance and impact artificially.