A spect CSE : Sentence Embeddings for Aspect-Based Semantic Textual Similarity Using Contrastive Learning and Structured Knowledge (2023)

Abstract: Generic sentence embeddings provide coarse-grained approximation of semantic textual similarity, but ignore specific aspects that make texts similar. Conversely, aspect-based sentence embeddings provide similarities between texts based on certain predefined aspects. Thus, similarity predictions of texts are more targeted to specific requirements and more easily explainable. In this paper, we present AspectCSE, an approach for aspect-based contrastive learning of sentence embeddings. Results indicate that AspectCSE achieves an average improvement of 3.97% on information retrieval tasks across multiple aspects compared to the previous best results. We also propose the use of Wikidata knowledge graph properties to train models of multi-aspect sentence embeddings in which multiple specific aspects are simultaneously considered during similarity predictions. We demonstrate that multi-aspect embeddings outperform even single-aspect embeddings on aspect-specific information retrieval tasks. Finally, we examine the aspect-based sentence embedding space and demonstrate that embeddings of semantically similar aspect labels are often close, even without explicit similarity training between different aspect labels.

## introduction sentence embeddings are representations of sentences or short text paragraphs in a dense vector space, such that similar sentences are close to each other @xcite . learning sentence embeddings is a fundamental task in natural language processing (nlp) and has already been extensively investigated in the literature @xcite @xcite @xcite @xcite . generic sentence embeddings can be used to distinguish between similar and dissimilar sentences, without considering which aspects of sentences are similar @xcite . moreover, they are often evaluated on generic semantic textual similarity (sts) tasks @xcite @xcite @xcite @xcite in which sentence similarity scores rely on human annotations. however, the concept of generic sts is not well defined, and text similarity depends heavily on the aspects that make them similar @xcite @xcite . we follow the argument of @xcite on textual similarity and define aspects as inherent properties of texts that must be considered when predicting their semantic similarity. based on the different aspects focused on in texts, their similarities can be perceived very differently. figure 1 illustrates an example of aspectbased sts. for example, wikipedia introduction texts of famous individuals can generally be considered similar as all texts introduce people who are known to the public. however, focusing the comparison on specific aspects (e.g., country of birth or profession) leads to different semantic similarity assessments for the same texts. although wikipedia is a special case as the introduction texts represent specific entities, this characteristic can nevertheless be generalized to different aspects found in any text. when deciding the similarity of texts, different aspects must be considered. consequently, human-annotated sts datasets introduce considerable subjectivity regarding the evaluated aspects. prior work uses siamese networks and a multiple negative ranking loss @xcite with only positive samples from the train set to create sentence embeddings for single aspects @xcite . sentence embeddings for single aspects only consider one specific aspect during similarity comparisons. using structured knowledge from knowledge graphs (kgs) for language model training has been shown to improve performances on all types of downstream tasks (schneider @xcite and also provides the possibility to create sentence embeddings that focus on multiple specific aspects simultaneously. these sentence embeddings are especially useful in information retrieval or unsupervised text classification settings @xcite @xcite . in this work, we advance state-of-the-art sentence embeddings for aspect-based sts using aspectcse, an approach for aspect-based contrastive learning of sentence embeddings. additionally, we introduce multi-aspect sentence embeddings that simultaneously consider multiple specific aspects during similarity comparisons. we show the effectiveness of multi-aspect sentence embeddings for both information retrieval and exploratory search tasks. finally, we demonstrate that using kg properties can be extremely beneficial for creating both single-and multi-aspect sentence embeddings. ## related work in nlp, aspects are most commonly examined in sentiment analysis problems @xcite @xcite @xcite . thus, the goal is to identify the aspects of given target entities and the sentiment expressed for each aspect @xcite . some works investigate aspect-based sts by considering it as a segmentation task. @xcite first segmented abstracts of research papers according to different aspects. then, they constructed semantic representations from these aspectbased segments, which can be used to find analogies between research papers. @xcite presented a human-annotated dataset that segments 10,966 english abstracts in the covid-19 open research dataset @xcite ) by the aspects background, purpose, method, result/contribution, and others. @xcite learned multivector representations of segmented scientific articles in which each vector encodes a different aspect. however, segmenting texts can harm their coherence and decrease the performance of downstream nlp models @xcite . other approaches propose to treat aspect-based sts as a pairwise multi-class classification problem @xcite . however, @xcite argue that pairwise classification with transformer models results in quadratic complexity. therefore, this approach is not suitable for large-scale sts tasks. to address the issues using previous approaches, @xcite proposed training aspectbased embeddings for research papers. in this work, we use aspectcse and kg properties to train single-and multi-aspect sentence embeddings. this allows us to focus on multiple specific aspects simultaneously while improving the performance of aspect-based sentence embeddings in sts tasks. 3 embedding methods ## data for our experiments, we use two different datasets. first, we use a benchmark dataset derived from papers with code (pwc) foot_0 to evaluate the effectiveness of aspectcse. we also use wikipedia and the wikidata kg @xcite to build a dataset for learning multi-aspect sentence embeddings. in all our experiments, we consider a pair of texts as positive if they share the same label for a particular aspect. accordingly, negatives comprise a pair of texts with different labels for a particular aspect. ## conclusion in this work, we proposed using wikidata knowledge graph properties to train single-aspect and multi-aspect sentence embedding models. unlike single-aspect sentence embeddings, multi-aspect sentence embeddings consider multiple specific aspects simultaneously during similarity comparisons. we regarded sts as an information retrieval task and introduced the aspectcse approach for training aspect-based sentence embedding models that achieve state-of-the-art performance on the pwc benchmark dataset. furthermore, we demonstrated that training aspect-based sentence embedding models on multiple aspects simultaneously even surpasses the performance of single-aspect sentence embedding models. finally, we show that the semantic similarity between different aspect labels is often connected to spatial proximity in the embedding space. this behavior is even clear if we train sentence embedding models only for similarity within the same aspect label but not explicitly for similarity between different aspect labels.