Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents (2023)

Abstract: Automatically highlighting words that cause semantic differences between two documents could be useful for a wide range of applications. We formulate recognizing semantic differences (RSD) as a token-level regression task and study three unsupervised approaches that rely on a masked language model. To assess the approaches, we begin with basic English sentences and gradually move to more complex, cross-lingual document pairs. Our results show that an approach based on word alignment and sentence-level contrastive learning has a robust correlation to gold labels. However, all unsupervised approaches still leave a large margin of improvement.

## introduction a pair of documents can have semantic differences for a variety of reasons: for example, one document might be a revised version of the second one, or it might be a noisy translation. highlighting words that contribute to a semantic difference is a challenging task (figure 1 ). previous work has studied word-level predictions in the context of interpretable textual similarity @xcite or evaluation of generated text @xcite , but did not necessarily focus on semantic differences as the main target. in this paper, we conceptualize the task of recognizing semantic differences (rsd) as a semantic diff operation. we assume that there are relatively few semantic differences and that many words are negative examples. our goal is to label words using self-supervised encoders such as xlm-r @xcite without additional training data. specifically, we investigate three simple metrics: figure 1 : while diff is a common tool for comparing code, highlighting the semantic differences in natural language documents can be more challenging. in this paper, we evaluate on synthetic documents that combine several such challenges, including cross-lingual comparison and non-monotonic sentence alignment. our experiments show that the first metric correlates best to the gold labels, since measuring the alignability of words has a relatively consistent accuracy across complexity levels. however, while unsupervised approaches have the advantage of not requiring manual annotations, we find that there is a considerable gap to perfect accuracy, especially for cross-lingual document pairs. future work could tackle the task by developing supervised models. besides providing a baseline, unsupervised metrics could also serve as features for such models. ## task formulation the goal of rsd is to analyze two word sequences @xmath0, . . . , a n and @xmath1, . . . , b m and to estimate, individually for each word, the degree to which the word causes a semantic difference between a and b. for example, given the sentences 'nice sweater!' and 'great news!', the correct labels would be close to 0 for 'nice' and 'great' and close to 1 for 'sweater' and 'news'. transformer-based encoders usually process documents as sequences of subword tokens. our strategy is to predict labels for individual subwords and to average the labels of the subword tokens that make up a word. to make the notation more readable, we use a and b to refer to the tokenized sequences as well. ## recognition approaches a simple approach to rsd is to calculate a soft token alignment between h(a) and h(b) and to identify tokens that are aligned with low confidence. a greedy alignment is usually calculated using the pairwise cosine similarity between hidden states @xcite . the prediction for a token a i is then given by: deletability of a word previous work has shown that encoders such as xlm-r may be fine-tuned such that the averages of their hidden states serve as useful sentence representations (reimers and gurevych, 2019; @xcite . the similarity of two sentences can be estimated using the cosine similarity between these averages: we approximate the similarity of a partial sequence a \ a i , where a i is deleted, by excluding the token from the average: the change in similarity when deleting a i can then serve as a prediction for a i , which we normalize to the range [0, 1]: cross-entropy of a word encoders such as xlm-r or bert @xcite have been trained using masked language modeling, which can be leveraged for our task. let h(a i |a ′ ) be the cross-entropy under a masked language model that predicts the token a i given a context a ′ , where a i has been masked. by concatenating b and a ′ into an augmented context ba ′ , we can test whether the additional context helps the language model predict a i . if the inclusion of b does not reduce the cross-entropy, this could indicate that b does not contain any information related to a we base our score on normalized pointwise mutual information (npmi), with a simple transformation to turn it into diff mask , a semantic difference score between 0 and 1. ## experimental setup we concatenate the 'headlines' and 'images' subsets of the ists dataset into a single dataset. we create a validation split by combining the ists training split with the paws-x validation split. similarly, we create a test split by combining the ists test split with the paws-x test split. appendix d reports data statistics. we perform our experiments on the multilingual xlm-r model of size 'base'. in addition to the standard model, we also evaluate a model finetuned on simcse @xcite . simcse is a self-supervised contrastive learning objective that is commonly used for adapting a masked language model to the task of embedding sentences. to train xlm-r with simcse, we use 1m sentences from english wikipedia and calculate sentence embeddings by averaging the final hidden states of the encoder. we train 10 checkpoints with different random seeds and report average metrics across the checkpoints. details are provided in appendix b. ## results finally, cross-lingual document pairs clearly present a challenge to all approaches, since we observe a consistent decline in terms of the average correlation across the six language pairs. appendix g provides results for the individual target languages, which show that comparing english documents to documents in japanese, korean or chinese is particularly challenging. appendices h and i juxtapose monolingual cross-lingual comparisons, illustrating that the latter are less accurate. 8th layer yield a more useful word alignment than the last layer, which confirms previous findings @xcite . interestingly, we find that fine-tuning the model with simcse strongly improves these results. even though simcse is an unsupervised sentence-level objective, the results suggest that learning sentencelevel representations also improves the quality of word alignment. this is in line with a related finding of leiter (2021) that supervised fine-tuning on nli can improve the explainability of bertscore. discussion of diff del relying on the deletability of a word has a lower accuracy than word alignment. in appendix f we test more complex formulations of diff del and find that accuracy can be improved by deleting bigrams and trigrams in addition to subword unigrams, but does not reach the accuracy of diff align . discussion of diff mask using the cross-entropy of masked language modeling yields some competitive results on longer documents. however, latency measurements show that this approach is much slower than the other approaches, since the documents need to be re-encoded for each masked word (appendix c). for the best-performing approach, diff align with simcse, we report results on the test split in appendix e. the test results confirm the patterns we have observed on the validation set. ## related work the idea of highlighting semantic differences or similarities on the level of individual words has influenced several research areas, notably interpretable semantic textual similarity @xcite and the evaluation of generated text @xcite @xcite . other application areas include content synchronization across documents @xcite and the detection of relevant differences in legal texts @xcite . some previous work has explored unsupervised or indirectly supervised approaches to such tasks. leiter (2021) measured the alignability of words to identify words that negatively impact the quality of translations. word alignment has also been used to estimate sentence similarity @xcite , and @xcite fine-tuned such a similarity metric with sentencelevel supervision in order to promote word-level interpretability. deleting words to measure their effect on sentence similarity is related to occlusion-based feature attribution methods @xcite . @xcite used a similar method to evaluate sentence representations. finally, the effect of context on cross-entropy (crossmutual information) has previously been analyzed in the context of machine translation @xcite . ## conclusion we formulated the task of recognizing semantic differences (rsd) between two documents as a token-level regression task and analyzed several unsupervised approaches towards this task. our experiments use annotations from ists, which we programmatically recombined into more challenging variants of the test set. we found that the alignability of a word is the most accurate measure, especially when the word alignment is computed with a simcse-adapted masked language model.