Contextualized Semantic Distance between Highly Overlapped Texts (2023)

Abstract: Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. Better evaluation of the semantic distance between the overlapped sentences benefits the language system’s understanding and guides the generation. Since conventional semantic metrics are based on word representations, they are vulnerable to the disturbance of overlapped components with similar representations. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence (LCS) as neighboring words and use masked language modeling (MLM) from pre-trained language models (PLMs) to predict the distributions in their positions. Our metric, Neighboring Distribution Divergence (NDD), represents the semantic distance by calculating the divergence between distributions in the overlapped parts. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts. Based on the discovery, we further implement an unsupervised and training-free method for text compression, leading to a significant improvement on the previous perplexity-based method. The high compression rate controlling ability of our method even enables NDD to outperform the supervised state-of-the-art in domain adaption by a huge margin. Further experiments on syntax and semantics analyses verify the awareness of internal sentence structures, indicating the high potential of NDD for further studies.

## introduction comparison between highly overlapped sentences exists in many natural language processing (nlp) tasks, like text rewriting @xcite and semantic textual similarity @xcite . a reliable evaluation of these paired sentences will benefit controllable generation and precise semantic difference understanding. conventional metrics, like the cosine similarity (s c ), have been popular for semantics similarity evaluation. nevertheless, we find the evaluating capability of s c severely degrades when the overlapping ratio rises. @xcite to introduce the difference between perplexity (∆ppl) to describe the semantic distance. unfortunately, ∆ppl suffers from the word frequency imbalance. also, many sentences share a similar ppl. based on the failure of s c , we hypothesize that the evaluation is disturbed by the overlapped components, which share similar representations in the paired sentences. we thus intend to mitigate the disturbance and thus propose a mask-and-predict strategy to attenuate the disturbance from overlapped words. compared to directly using the word representations for comparison, we discover that using predicted distributions from masked language modeling (mlm) results in better evaluation. taking figure 1 as the instance, unmasked comparison involves similar representations of the and heavy in both sentences since the encoder can see these words and encode with their information. but pre-trained language model my life will be mine . my life will be mine . my life be mine . when these words are masked, the mlm has to predict the distributions considering the contextual difference. while using representations results in a trivial heavy-heavy comparison, the difference of distributions (between candidates long, short and large,wide) better indicates how the contextual semantics changes. thus, we are motivated to propose a new metric, neighboring distribution divergence, which compares predicted mlm distributions from pretrained language models (plms) and uses the the divergence between them to represent the semantic distance. we take the overlapped words in the longest common sequence (lcs) between the paired sentences as neighboring words for the divergence calculation. we conduct experiments on semantic textual similarity and text compression. experiment results verify ndd to be more sensitive to precise semantic differences than conventional metrics like s c . experiments on the google dataset show our method outperforms the previous ppl-based baseline by around 10.0 on f1 and rouge scores. moreover, the ndd-based method enjoys outstanding compression rate controlling ability, which enables it to outperform the supervised state-of-the-art by 18.8 f1 scores when adapting to broad news compression corpus in a new domain. the cross-language generality of ndd is also verified by experiments on a chinese colloquial sentence compression dataset. we further use syntax and semantics analyses to test ndd's awareness of the sentence's internal structure. our experiments show that ndd can be applied for accurate syntactic subtree pruning and semantic predicate detection. results from our analyses verify the potential of ndd on more syntax or semantics-related tasks. our contributions are summarized as follows: ## semantic distance evaluation we conduct experiments on the test dataset of semantic textual similarity benchmark foot_0 (sts-b) to analyze the metrics. multiple sentence pair similarity evaluation tasks are designed to compare the metric performance and investigate the metric property. we replace 20% words for synonym-antonym, pos, and lemma tests. 100% verbs are replaced for term tests. the words for the replacement are sampled from the sts test dataset following their frequency. for the supervised test, we sample sentence pairs with an lcs that consists of at least 80% words in the shorter sentence. we use roberta base as the plm and apply hellinger distance as the divergence function to guarantee the boundary of our metric. mean pooling is used as the attention-assigning strategy. the sts experiment results are presented in figure 3 shows how the ratio of overlapped words affects the metric performance. @xmath0, so we are only ## unsupervised text compression the prominent performance of ndd and its correlation with overlapped word ratio inspire us to apply it for extractive text compression. text compression takes a sentence w as the input and outputs w c where w c is a substring of w that maintains the main semantics in w . as a substring, the compressed sentence guarantees a 100% overlapped ratio to support ndd's performance. ## further analysis 8 we continue studying the compression algorithm to further investigate ndd's syntax awareness via analyzing the roles of pruned words in the syntax treebank. ## related works the evaluation on text similarity provides valuable guidance on various downstream tasks, including text classification @xcite , document clustering @xcite , and translated text detection @xcite . the commonly used cosine similarity evaluates paired sentences' similarity based on the cosine value between word embeddings or pre-trained representations @xcite . unfortunately, when the overlapping ratio between paired sentences rises, the representationbased method suffers from faults caused by similar word representations. our work replaces word representations with predicted distributions to mitigate the disturbance from overlapped components. the proposal of plms @xcite inspires researchers to leverage the upstream training process for text similarity evaluation. niu et al. leverage the perplexity calculated from plms to represent the semantic distance between texts during text compression. while perplexity can evaluate the fluency of sentences, a recent study @xcite suggests that low perplexity does not directly refer to a human-like sentence. also, perplexity fails with words that share a similar existing probability but are with opposite or irrelevant meanings. other plm-based metrics like bertscore have been verified by experiments to evaluate text generation better @xcite . other pre-trained models for evaluation are also an interesting topic. to evaluate semantics preservation in amr-to-sentence, opitz and frank exploits amr parser to compare the amr graph of generated results with the golden graph, showing the potential of pre-trained models to evaluate more complex linguistic structures. many supervised methods @xcite have been proposed for text compression. syntax treebanks play a critical role in text compression @xcite @xcite . unsupervised methods have been explored to extract sentences from documents to represent key points @xcite . nevertheless, span pruning is still far from satisfaction. as mentioned before, @xcite explores using ∆ppl for compression, which is not so capable as ndd in semantics preservation. syntax and semantic analyses @xcite ) reflect model's awareness of the internal structures in sentences. the awareness of syntax and semantics of ndd is verified by those tasks. ## conclusion we address the overlapping issue in semantic distance evaluation in this paper. to mitigate the disturbance from overlapped components, we mask and predict words in the lcs via plm-based mlm. ndd evaluates the semantic distance using a weighted sum of the divergence between predicted distributions. sts experiments verify ndd to be more sensitive to a wide range of semantic differences and perform better on highly overlapped paired texts, which is challenging for conventional metrics. ndd-based text compression algorithm significantly boosts the unsupervised performance, and its high compression rate controlling ability enables the adaption to datasets in different domains. ndd's awareness of syntax and semantics is verified by further analyses, showing the potential of ndd for further studies.