Improving word mover’s distance by leveraging self-attention matrix (2023)

Abstract: Measuring the semantic similarity between two sentences is still an important task. The word mover’s distance (WMD) computes the similarity via the optimal alignment between the sets of word embeddings. However, WMD does not utilize word order, making it challenging to distinguish sentences with significant overlaps of similar words, even if they are semantically very different. Here, we attempt to improve WMD by incorporating the sentence structure represented by BERT’s self-attention matrix (SAM). The proposed method is based on the Fused Gromov-Wasserstein distance, which simultaneously considers the similarity of the word embedding and the SAM for calculating the optimal transport between two sentences. Experiments demonstrate the proposed method enhances WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity.

## introduction the task of measuring the semantic textual similarity (sts) of two sentences is essential for natural language processing with various applications @xcite . there are several methods for measuring sts, among which many methods using optimal transport (ot) distance have been proposed and have shown good performance @xcite @xcite . ot theory gives a method to measure the difference between two distributions by setting the transport cost of a unit mass and considering the allocation of the transported mass to minimize the total cost. this allocation is called optimal transport, and the total cost is called the ot distance. a basic form of ot distance is the wasserstein distance, which measures the similarity between sets using the distance between the elements in figure 1 : an illustration of ot for word embeddings from sentence 1 to sentence 2. words are aligned by word similarity in wmd; e.g., obama matches president. words are aligned by sentence structure in smd or by word similarity and sentence structure simultaneously in wsmd; e.g., obama matches press. see section 2. the sets @xcite . word mover's distance (wmd) computes wasserstein distance by considering a sentence as a set of word embeddings @xcite . however, wmd does not consider the word order of sentences, making it challenging to identify paraphrases. let us see the following illustrative example of the paraphrases adapted from @xcite and @xcite : here (b) is a paraphrase of (a), while (c) has a very different meaning from (a). however, all these sentences have a high overlap of words. thus, wmd cannot distinguish these pairs. to account for sentence structure, we focused on bert @xcite , an attention-based model that has recently achieved remarkable performance in natural language processing tasks. bert is a masked language model that performs taskspecific fine-tuning after pre-training on a large 1 ). the difference of sam values is small for the word alignment of smd or wsmd; e.g., (obama, speaks) matches (press, greets). the difference is large for that of wmd; e.g., (obama, speaks) matches (president, greets). see section 2. we propose a novel method called sentence structure mover's distance (smd) in section 5 that measures the optimal transport distance between sentence structures represented in sams. smd aims to find the optimal word alignment by considering the transport cost between elements of sams, while wmd considers the transport cost between word embeddings. to compute smd, we employ the gromov-wasserstein (gw) distance @xcite , the optimal transport distance for measuring structural similarity between sets. smd is ineffective by itself because it measures only the difference between sentence structures. thus smd is used in combination with wmd. we improve wmd-like methods (i.e., wmd and its variants based on wasserstein distance between sets of word embeddings) by combining smd with them in section 6. this proposed method is called word and sentence structure mover's distance (wsmd). wsmd simultaneously measures the difference between word embeddings and the difference between sentence structures. wsmd employs the fused gromov-wasserstein (fgw) distance @xcite , known as the optimal transport distance that combines both wasserstein and gw distances. ## illustrative example here we explain how our proposed method works for the example of the paraphrases in section 1. the original wmd and its improvement with our proposal are computed for this example and shown in looking at the values of the original wmd for the two cases, we find that they are about the same. thus, wmd failed to give a reasonable sentence similarity. this failure is explained in fig. 1 by illustrating word embeddings for case (2). wmd tries to find the closest word for each word, matching obama to president, media to press, and so on; the word alignment is indicated as the short dotted arrows. then, wmd is computed by averaging the length of these short arrows, indicating a high sentence similarity contrary to the fact that the actual similarity is low for case (2). however, the word alignment for wmd is inappropriate, given the word order and sentence structure. our proposed method correctly matches obama to press and media to president. this word alignment is shown as the long arrows with solid lines in fig. 1 . wmd λ in fig. 2 explains how our proposed method takes account of the sentence structure. given a word alignment between two sentences, we can think of a matching of elements between the two sams by applying the word alignment to both the rows and the columns of the matrices. for example, the word alignment of wmd in fig. 1 induces the matching from (obama, speaks) to (president, greets), (media, speaks) to (press, greets), and so on. smd minimizes the average difference of sam values case sentences wmd wmd λ wsmd similarity (1) (a) obama speaks to the media in illinois. 12.54 12.54 7.26 high (b) the president greets the press in chicago. (2) (a) obama speaks to the media in illinois. 12.55 13.30 8.03 low (c) the press greets the president in chicago. ## related work this paper focuses on sentence similarity measures using the ot of word embeddings. for example, wmd @xcite uses uniform word weights and the l 2 distance of word embeddings as the transport cost. by modifying the word weights and the transport cost, various wmd-like methods have been developed to compute sentence similarity based on the wasserstein distance between sets of word embeddings. for example, word rotator's distance (wrd) @xcite uses the vector norm of the word embedding for the weight and the cosine similarity for the cost. there are many attempts to improve wmd by incorporating word order and sentence structure information into the weight, cost, and penalty terms, as explained below. however, none of them consider the optimal transport of sentence structures nor utilize sams from bert. in opwd @xcite , the temporal difference between word embeddings w i and w ′ j is measured by the distance between their relative temporal positions (i/nj/m) 2 , where n and m represent the lengths of the corresponding sentences. this difference is incorporated into the transport cost and an additional regularization term. wmdo @xcite identifies consecutive words common to two sentences as a chunk and introduces a penalty term according to the number of chunks. in syntax-aware wmd (synwmd) @xcite , the word weight is computed from the word co-occurrence extracted from the syntactic parse tree of sentences, and a word embedding incorporates those from its subtree of the parse tree. the weight and the cost in synwmd are called syntax-aware word flow (swf) and syntax-aware word distance (swd), respectively. in recursive optimal transport similarity (rots) @xcite , a recursive optimal transport method is introduced to capture word dependencies in sentences. to measure sentence similarity, the pairwise cosine similarity of words is weighted using this optimal transport and aggregated. in mover-score @xcite , the inverse document frequency (idf) is used for the word weight, and the word embeddings from bert are used for computing the cost. bertscore @xcite also uses idf and bert embedding, but it employs greedy matching instead of ot for word alignment. ## optimal transport of words we review the computation of wmd. in the following, we denote two sentences as the weights on w i and w ′ j in the probability distributions for s and s ′ are denoted by @xmath0, with n @xmath1, and u ′ = (u ′ j ) m @xmath2, with m @xmath3, respectively. in this paper, unless otherwise stated, the weights on words in a sentence are equal, i.e., the uniform distribution specified as ## optimal transport of sentence structures in this section, we propose to apply the gw distance to the sam of bert to measure the optimal transport distance between sentence structures. later in section 6, we will attempt to improve wmd using this method. ## optimal transport of words and sentence structures here, we propose an optimal transport distance that simultaneously considers the word embeddings and sentence structure. ## experiments we compare the performance of our proposed method and existing baseline methods on the task of measuring semantic textual similarity (sts) between two sentences. ## conclusion since wmd treats a sentence as a set of word embeddings and computes sentence similarity, it cannot consider the word order in the sentence. therefore, we focused on the fact that the sam of the input sentence obtained from the pre-trained bert represents the relationship between words in the sentence and has information on the sentence structure. we proposed an optimal transport distance wsmd that improves existing wmd-like methods by using fgw distance that simultaneously measures the difference between word embeddings and the difference between sentence structures. we conducted experiments on paraphrase identification on paws and sentence similarity on stsb, confirming the proposed method boosts paws performance with minimal impact on stsb.