Sentence Similarity Based on Contexts (2022)

Abstract: Existing methods to measure sentence similarity are faced with two challenges: (1) labeled datasets are usually limited in size, making them insufficient to train supervised neural models; and (2) there is a training-test gap for unsupervised language modeling (LM) based models to compute semantic scores between sentences, since sentence-level semantics are not explicitly modeled at training. This results in inferior performances in this task. In this work, we propose a new framework to address these two issues. The proposed framework is based on the core idea that the meaning of a sentence should be defined by its contexts, and that sentence similarity can be measured by comparing the probabilities of generating two sentences given the same context. The proposed framework is able to generate high-quality, large-scale dataset with semantic similarity scores between two sentences in an unsupervised manner, with which the train-test gap can be largely bridged. Extensive experiments show that the proposed framework achieves significant performance boosts over existing baselines under both the supervised and unsupervised settings across different datasets.

## introduction measuring sentence similarity is a long-standing task in nlp @xcite @xcite . the task aims at quantitatively measuring the semantic relatedness between two sentences, and has wide applications in text search @xcite , natural language understanding (maccartney and manning, 2009), and machine translation @xcite . one of the greatest challenges that existing methods face for sentence similarity is the lack of large-scale labeled datasets, which contain sentence pairs with labeled semantic similarity scores. the acquisition of such a dataset is both labor-intensive and expensive. for example, the sts benchmark @xcite and sick-relatedness dataset @xcite respectively contain 8.6k and 9.8k labeled sentence pairs, the sizes of which are usually insufficient for training deep neural networks. unsupervised learning methods are proposed to address this issue, where word embeddings @xcite or bert embeddings @xcite are used to to map sentences to fix-length vectors in an unsupervised manner. then sentence similarity is computed based on the cosine or dot product of these sentence representations. our work follows this thread where sentence similarity is computed based on fix-length sentence representations, as opposed to comparing sentences directly. the biggest issue with current unsupervised approaches is that there exists a big gap between model training and testing (i.e., computing semantic similarity between two sentences). for example, the bert-style models are trained at the token level by predicting words given contexts, and there is neither explicit modeling sentence semantics nor producing sentence embeddings at the training stage. but at test time, sentence semantics needs to be explicitly modeled to obtain semantic similarity. the inconsistency results in a distinct discrepancy between the objectives at the two stages and inferior performance on textual semantic similarity tasks. for example, bert embeddings yield inferior performance on semantic similarity benchmarks @xcite , and even underperform the naive method such as averaging glove @xcite embeddings. @xcite investigated this problem and found that bert always induces a non-smooth anisotropic semantic space of sentences, and this property significantly harms the performance of semantic similarity. just as word meanings are defined by neighboring words @xcite , the meaning of a sentence is determined by its contexts. given the same context, there is a high probability of generating two similar sentences. if there is a low probability of generating two sentences given the same context, there is a gap between these two sentences in the semantic space. based on this idea, we propose a framework that measures semantic similarity through the probability similarity of generating two sentences given the same context in a fully unsupervised manner. as for implementation, the framework consists of the following steps: (1) we train a contextual model by predicting the probability of a sentence fitting into the left and right contexts; (2) we obtain sentence pair similarity by comparing scores assigned by the contextual model across a large number of contexts. to facilitate inference, we train a surrogate model, to act as the role of step 2, based on the outputs from step 1. the surrogate model can be directly used for sentence similarity prediction in an unsupervised setup, or used as initialization to be further finetuned on downstream datasets in the supervised setup. note that the outcome from step 1 or the surrogate model is a fixed-length vector regarding the input sentence. each element in the vector indicates how fit the input sentence is to the context corresponding to that element, and the vector itself can be viewed as the overall semantics of the input sentence in the contextual space. then we use cosine distance between two sentence vectors to compute the semantic similarity. the proposed framework offers the potential to fully address the two challenges above: (1) the context regularization provides a reliable means to generate a large-scale high-quality dataset with semantic similarity scores based on unlabeled corpus; and (2) the train-test gap can be naturally bridged by training the model on the largescale similarity dataset, leading to significant performance gains compared to utilize pretrained models directly. we conduct experiments on different datasets under both supervised and unsupervised setups, and experimental results show that the proposed framework significantly outperforms existing sentence similarity models. ## related work statistics-based methods for measuring sentence similarity include bag-of-words (bow) @xcite , term frequency inverse document frequency (tf-idf) @xcite , bm25 @xcite , latent semantic indexing (lsi) @xcite , and latent dirichlet allocation (lda) @xcite . deep learning based methods for sentence similarity rely on distributed representations @xcite and can be generally divided into the following three categories. ## ablation studies we perform comprehensive ablation studies on the stsb dataset with no additional training on nli datasets to better understand the behavior of ## conclusion in this work, we propose a new framework for measuring sentence similarity based on the fact that the probabilities of generating two similar sentences based on the same context should be similar. we propose a pipelined system by first harvesting massive amounts of sentence pairs along with their similarity scores, and then training a surrogate model using the automatically labeled sentence pairs for the purpose of faster inference. extensive experiments demonstrate the effectiveness of the proposed framework against existing sentence embedding based methods.