Going Beyond Sentence Embeddings: A Token-Level Matching Algorithm for Calculating Semantic Textual Similarity (2023)

Abstract: Semantic Textual Similarity (STS) measures the degree to which the underlying semantics of paired sentences are equivalent. State-of-the-art methods for STS task use language models to encode sentences into embeddings. However, these embeddings are limited in representing semantics because they mix all the semantic information together in fixed-length vectors, which are difficult to recover and lack explainability. This paper presents a token-level matching inference algorithm, which can be applied on top of any language model to improve its performance on STS task. Our method calculates pairwise token-level similarity and token matching scores, and then aggregates them with pretrained token weights to produce sentence similarity. Experimental results on seven STS datasets show that our method improves the performance of almost all language models, with up to 12.7% gain in Spearman’s correlation. We also demonstrate that our method is highly explainable and computationally efficient.

## introduction measuring the similarity between two sentences is an important task in many natural language processing (nlp) applications. this makes semantic textual similarity (sts) a crucial preliminary step in various domains, such as information retrieval @xcite , machine translation @xcite , plagiarism detection @xcite , semantic search @xcite , and conversational systems @xcite . large pretrained language models @xcite have achieved the stateof-the-art performance on sts task @xcite @xcite . these approaches typically use language models to encode input sentences into embeddings and then calculate sts using similarity metrics such as the cosine function. however, sentence embeddings have limitations in representing sentences, as all the information of the sentence is aggregated and mixed together in the fixed-length embedding. this problem is especially pronounced for the sts task, which requires fine-grained, low-level semantic understanding and comparison @xcite . as a result, methods based on sentence embeddings often have difficulty being well-trained and lack explainability for their predicted results. going beyond sentence embeddings, we propose a token-level matching algorithm for sts. our algorithm works in the inference stage, so it can be applied on top of any trained language model to improve its performance. specifically, given a trained language model (also called base model), we use it to generate token embeddings for the two input sentences and calculate their pairwise token similarity. we then design a novel scoring function to calculate the matching score for each token. the sentence similarity score is calculated by averaging all the token matching scores with token weights, which are learned unsupervisedly from a large corpus. our method captures fine-grained, token-level information, which is more indicative, robust, and explainable than sentence embeddings. we conducted experiments on seven standard sts datasets using six language models and their variants as base models. our method is able to improve the performance of almost all existing language models, especially those "poor" ones (up to 12.7% improvement in spearman's correlation). specifically, our model improves simcse by 0.8% to 2.2%, and improves esimcse by 0.6% to 1.2%, which is the current state-of-the-art model on the sts task. we also demonstrated the explainability of our model by identifying the semantically similar parts between two input sentences. ## related work existing work on sts can be broadly divided into two categories: lexicon-based and semanticbased. lexicon-based approaches @xcite @xcite calculate the correlation between the character streams of two sentences being compared, which can be applied at the level of characters or words. semantic-based approaches can be further divided into three categories: word-based methods @xcite , which treat a sentence as a list of words and compare the correlations between words; structure-based methods, which use language tools such as grammar @xcite , part-of-speech @xcite , and word order @xcite to process sentences and compare their structure; and vector-based methods @xcite @xcite @xcite , which calculate sentence embeddings that describe each sentence as a vector and have achieved the state-of-the-art performance on sts. our method is conceptually similar to bert-score @xcite , a token-level evaluation metric for text generation. however, there are two significant differences between these two approaches: (1) bertscore is an evaluation metric, while our method is an algorithm for calculating sts; ## the proposed method given a pair of two sentences @xmath0, t 2 , • • • , t |s| ⟩ and ŝ = ⟨ t1 , t2 , • • • , t|ŝ| ⟩ where t i ( ti ) is the i-th token in sentence s (ŝ), our goal is to learn a function f (s, ŝ) ∈ r that calculates the semantic similarity between s and ŝ. token-level similarity matrix we can calculate token embeddings for s and ŝ using any language model, including pretrained language models @xcite , or language models specifically finetuned for the sts task @xcite @xcite . given sentence s and ŝ, the language model generates the token embedding matrix @xmath1, where each row corresponds to a d-dimension token embedding. the token-level similarity matrix for s and ŝ is then calculated as @xmath2, in which the entry s ij indicates the similarity between token t i and tj . token matching score the token matching score measures the likelihood that a given token in one sentence can be matched to a token in the other sentence. this score takes into account two aspects: the token matching score is defined as the sum of the above two scores: token weighting tokens typically have different levels of semantic importance. previous work @xcite uses inverse document frequency (idf) as token weights, as rare words can be more indicative than common words. however, in many cases, high-frequency words can be semantically important (e.g., "not") while low-frequency words may be semantically unimportant (e.g., specific numbers). to address the mismatch between token importance and token frequency, we propose learning token weights from plain texts. specifically, we choose unsupervised simcse @xcite as the training model, which takes an input sentence and predicts itself in a contrastive objective with dropout used as noise. during the training stage of simcse, instead of using the lastlayer embedding of cls token as the sentence embedding, we assign a trainable weight parameter w i for each token i in the vocabulary and calculate the weighted average @xmath3, where t i is the last-layer embedding of token @xmath4. in this way, the token weights w can be trained together with the model parameters of simcse on a large unsupervised corpus, which has been shown to be more semantically precise than frequency-based token weights. the final sts score of the input sentences (s, ŝ) is the weighted average of all token scores: ## conclusion this paper presents a token-level matching algorithm for calculating sts between pairs of sentences. unlike previous approaches that use pretrained language models to encode sentences into embeddings, our method calculates the pairwise token similarity, and then applies a token matching functions to these scores. the resulting scores are averaged with pretrained token weights to produce the final sentence similarity. our model consistently improves the performance of existing language models and is also highly explainable, with minimal extra time overhead during inference.