I love pineapple on pizza != I hate pineapple on pizza: Stance-Aware Sentence Transformers for Opinion Mining (2024)

Abstract: Sentence transformers excel at grouping topically similar texts, but struggle to differentiate opposing viewpoints on the same topic. This shortcoming hinders their utility in applications where understanding nuanced differences in opinion is essential, such as those related to social and political discourse analysis. This paper addresses this issue by fine-tuning sentence transformers with arguments for and against human-generated controversial claims. We demonstrate how our fine-tuned model enhances the utility of sentence transformers for social computing tasks such as opinion mining and stance detection. We elaborate that applying stance-aware sentence transformers to opinion mining is a more computationally efficient and robust approach in comparison to the classic classification-based approaches.

## introduction sentence transformers have become a cornerstone of natural language processing (nlp), revolutionizing tasks like sentiment analysis, document retrieval, and text classification by capturing semantic meaning and contextual nuances. however, they grapple with a specific limitation that significantly impedes their utility in social computinga critical domain where understanding sociopolitical stances is vital (e.g. @xcite ). in social computing, opinion mining and stance detection tasks demand the ability to discern between sentences expressing opposing stances on the same topic @xcite . conventional sentence transformers often fall short in this regard, producing highly similar vectors even for sentences with contrasting opinions @xcite . for instance, the embeddings provided by the stateof-the-art sentence transformers for the sentences: "the weather is good" vs. "the weather is not good" manifest a high level of similarity in the embedding space, since both are talking about the quality of the weather, but with the exact opposite stance. in other words, they are topically similar, but stance-wise dissimilar. this limitation is a major obstacle in tasks related to controversial sociopolitical topics where identifying differing perspectives is essential. take, for instance, a situation where we want to automate the identification of the pro-and anti-abortion posts on twitter through semantic search or semantic clustering of the sentence embeddings @xcite . using the default sentence transformers would group both pro-and anti-abortion tweets together since they are merely similar topic-wise. this disables the semantic method from detecting the stances of certain twitter users with the automated and computationally cheap utilization of sentence transformers. an alternative, but computationally expensive, approach is to train a classifier capable of distinguishing the stances of pairs of statements @xcite @xcite . however, this would require inputting pairs of sentences into the model at each point of pairwise comparison, with a subpar complexity in the order of n 2 times for pairwise comparison of n statements. we address existing limitations by empowering sentence transformers, a computationally efficient method, with stance awareness. we extract and compose a rich dataset of supporting and opposing statements on controversial topics to fine-tune these models. our objective is to lessen cosine similarities for statements representing opposing stances and increase similarities for congruent viewpoints. we perform this by fine-tuning a state-of-the-art sentence transformer with siamese and triplet networks using a contrastive and triplet loss function on top of the networks. these loss functions penalize the model for providing spatially close embeddings for contradictory, yet topically similar, pairs (triplets) of text. in summary, our work makes the following con-tributions: ## motivation & related work the main objective of this work is to enhance opinion mining and stance detection tasks. thus, in this section, we motivate our work by examining the limitations of prior work. motivation: stance detection is a vital task in social computing, aiming to identify an author's viewpoint (e.g., in favor, against, neutral) towards a specific topic @xcite . existing methods leverage state-of-the-art nlp architectures, such as bert @xcite , to classify the semantic relationship between a target sentence and a context sentence expressing a known stance. moreover, recent advancements in llms, have demonstrated significant potential in performing various nlp tasks, including stance detection, in a zero-shot setting without the need for finetuning @xcite . however, both the supervised classificationbased and the llm-based approaches come with a significant computational cost. since they involve feeding both the target sentence and the context sentence into the model simultaneously, for n pieces of text, they require calling the model n 2 times. this can be particularly problematic when dealing with large datasets or real-time analyses, such as analyzing stances in social media streams containing millions of posts. for instance, feeding dot-separated pairs of sentences to bert-base to predict their relationship @xcite (e.g., predicting sim-ilarity, predicting stance), would take an average inference time of 32ms per sentence pair on nvidia tesla v100 gpu @xcite . comparing the stances of all the sentence pairs for 1,000 sentences will take 4.5 hours ≈ 32ms × 1000 2 . rise of sentence transformers: to address this problem of enormous computational workload for sentence similarity tasks, sentence transformers were introduced @xcite . by fine-tuning bert with siamese networks, reimers et al. proposed a way to generate semantically meaningful sentence embeddings that are spatially close for semantically similar sentences. these pre-generated embeddings removed the need for calling the models for every pairwise comparison, reducing the complexity to only n times for mapping the embeddings of n sentences; totaling: 32ms × n. then, the similarity of every sentence pair is obtained by a swift calculation of the spatial distance of their pre-generated embeddings (approx 0.5ms per vector pair distance calculation). thus, comparing all pairwise combinations for 1,000 sentences in terms of similarity would only take 4.5 minutes ≈ 32ms × 1000 + 0.5ms × 1000 2 . need for stance-aware sentence transformers: the sentence transformers can solve the problem of computational inefficiency in sentence similarity measurement. yet, if the task would be to compare the stances of sentence pairs on similar topics, current sentence transformers would perform far below ideal as they often confuse topic-wise similarity with stance-wise similarity; a limitation that has also been highlighted by previous work @xcite . this often results in assigning high similarity scores to statements that express opposing positions on the same topic. for example, "i love pineapple on pizza" and "i hate pineapple on pizza", two opposing stances on pizza, will be assigned a high similarity score as they are both talking about a taste towards the same food. another significant limitation of sentence transformers and similar models is their poor handling of negations and antonyms, as shown by recent research. @xcite demonstrate that sentence embeddings often fail to capture meaningpreserving transformations when one sentence includes a negated antonym of the other, such as "i am not guilty" and "i am innocent." this deficiency further exacerbates the challenge of stance detection, where subtle shifts in meaning can completely reverse the stance. developing the ability to fine-tune sentence transformers for spatial dissimilarity in opposing viewpoints has the potential to significantly advance online opinion mining and stance detection. take, as a running example, a case where we want to figure out the stances of several politicians on abortion rights using their twitter timelines. a solution aided by sentence transformers, as we demonstrate in §5.3, can query anti-and pro-abortion statements such as "abortion is murder" and "abortion is healthcare." then, after embedding both queries and timelines into vectors using sentence transformers, we can systematically infer tweets with high spatial similarity to the pro (anti) abortion query and their stance. another huge computational advantage of this approach is that the embeddings generated for the timelines can be saved and used for other queries in the future. for example, we can quickly generate a pair of queries representing pro-and anti-guncarrying rights and run them on the same timelines that are already vectorized to mine the users' opinions on gun control. ## methodology in this section, we elaborate on the fine-tuning architecture and our experimental settings for strategizing the fine-tuning process. figure 1 summarizes the entire pipeline of our approach, including fine-tuning ( §3), data-preparation ( §4), and the semantic-search application ( §5). ## experiments, results, & observations we next describe our experiments and results after applying our method to fine-tune the sentence transformer. we first test the performance of all the fine-tuned models on a test set from the kialo and sts-b datasets ( §5.1 and §5.2). using the best-performing model, we evaluate how the learning transfers to another dataset ( §a.5). finally, we showcase its application on semantic search for opinion mining ( §5.3). ## discussion this work tackles the critical challenge of balancing three essential requirements in nlp tasks: computational efficiency (r1), stance awareness (r2), and maintaining topic awareness (r3). we address these challenges by proposing a novel approach that leverages fine-tuning while mitigating its drawbacks. we reviewed how prior work fails to meet these three requirements together in §2 and we showed how our work ( §3) addresses them ( §5), we next summarize the main findings of our paper a discuss their implications and limitations. computational efficiency. our approach makes opinion mining efficient, only needing to call the model n times for mapping the embeddings of n sentences, that is, linear with the number of sentences. a limitation may arise in how much a single statement used as a query might encompass all variations of the stance on a certain topic. an important consideration is to maintain sufficient diversity in query selection to account for all parts of the spectrum of opinions. a balance is feasible. our work demonstrates the feasibility of achieving a balance between efficiency, stance awareness, and topic coherence through careful fine-tuning strategies. this approach can be further explored and adapted for various nlp applications, particularly those requir ## conclusion overall, our work paves the way for stance-aware sentence transformers, offering a powerful tool for social computing tasks like opinion mining. our work demonstrably surpasses the state-of-theart in stance awareness of sentence transformers, achieving significant improvements in distinguishing stances across in-distribution (kialo test-set) and out-of-distribution (semeval 2014 and twitter) datasets. by designing an innovative model architecture, we observed a measurable improvement of results with the hybrid (combination of siamese and triplet) model. we implemented a data filtering approach by removing low cosine similarity pairs, which probed a unique experimental contribution that effectively mitigated the impact of "lowquality" human-generated data within the training set. this also resulted in an improvement of the model performance, while significantly reducing the train-set size and thus the training time. two main future steps in this direction can significantly improve the quality of the task: 1) improving general-purpose sentence transformers using (llms) and extensive datasets, such as recently developed open ai's text embedders 4 ; 2) developing dedicated datasets tailored to social media plat-4 https://platform.openai.com/docs/guides/embeddings forms like twitter and mastodon and fine-tuning the general-purpose sentence transformer on such datasets. this will enable the model to learn stance awareness in the context of the targeted social networks of analysis. nevertheless, our model, which is fine-tuned on kialo arguments also demonstrated a promising performance on the twitter data. this forecasts an even brighter future for models that are specifically fine-tuned on online social media data for the same task. reproducibility: we open-source both code and models to foster reproducibility. 5