Sentence Representations via G aussian Embedding (2024)

Abstract: Recent progress in sentence embedding, which represents a sentence’s meaning as a point in a vector space, has achieved high performance on several tasks such as the semantic textual similarity (STS) task.However, a sentence representation cannot adequately express the diverse information that sentences contain: for example, such representations cannot naturally handle asymmetric relationships between sentences.This paper proposes GaussCSE, a Gaussian-distribution-based contrastive learning framework for sentence embedding that can handle asymmetric inter-sentential relations, as well as a similarity measure for identifying entailment relations.Our experiments show that GaussCSE achieves performance comparable to that of previous methods on natural language inference (NLI) tasks, and that it can estimate the direction of entailment relations, which is difficult with point representations.

## introduction sentence embeddings are representations to describe a sentence's meaning and are widely used in natural language tasks such as document classification @xcite , sentence retrieval @xcite , and question answering @xcite . in recent years, machine-learning-based sentence embedding methods with pre-trained language models have become mainstream, and various methods for learning sentence embeddings have been proposed @xcite . however, as these methods represent a sentence as a point in a vector space and primarily use symmetric measures such as the cosine similarity to measure the similarity between sentences, they cannot capture asymmetric relationships between two sentences, such as entailment and hierarchical relations. s1：my male friends are playing soccer. s2：some men are playing a sport. s3：the man is sleeping. to handle such asymmetric relationships between sentences by extending gaussian embedding for words @xcite . figure 1 shows examples of sentence representations obtained by a previous method and by gausscse. whereas the previous method represents a sentence as a point, gausscse represents a sentence as a region in the embedding space, and when two sentences have an entailment relation, the embedding of the entailing sentence contains the embedding of the entailed one. in these examples, s1 entails s2, but with previous methods, it is difficult to determine the entailment relation only from their embeddings. in contrast, by taking into account the variances of the distributions, gausscse can capture the asymmetric relationship where s1 entails s2 but s2 does not entail s1, as well as the fact that s3 is not in the entailment relationship with either s1 or s2. to validate the usefulness of gausscse, we performed comparative experiments on two tasks: the natural language inference (nli) task, and the task of predicting the entailment direction. the results demonstrate that gausscse can accurately predict the entailment direction while maintaining good performance on the nli task. foot_0 ## sentence representations via gaussian embedding gausscse is a method to obtain gaussian embeddings of sentences by fine-tuning a pre-trained language model through contrastive learning. in this section, we first review a representative study of gaussian embeddings and then review simcse, a method that acquires sentence embeddings via contrastive learning. we also review embedding methods that focus on asymmetric relations, which is closely related to our research. we then describe gausscse, which extends gaussian embeddings and simcse. ## experiments we validated the effectiveness of gausscse through experiments on two tasks: nli and prediction of the entailment direction. ## conclusion in this paper, we have presented gausscse, a gaussian-distribution-based contrastive sentence embedding to handle asymmetric relationships between sentences. gausscse fine-tunes pre-trained language models via contrastive learning with asymmetric similarity. through experiments on the nli task and entailment direction prediction, we have demonstrated that gausscse achieves comparative performance to previous methods on nli task and also accurately estimate the direction of entailment relations, which is difficult with conventional sentence representations. in this study, we used a gaussian distribution to represent the spread of the meaning of a sentence in the embedding space, we would like to conduct a comparison with the use of other types of embedding, such as hyperbolic embeddings @xcite or box embeddings @xcite in future work.