Towards Structure-aware Paraphrase Identification with Phrase Alignment Using Sentence Encoders (2022)

Abstract: Previous works have demonstrated the effectiveness of utilising pre-trained sentence encoders based on their sentence representations for meaning comparison tasks. Though such representations are shown to capture hidden syntax structures, the direct similarity comparison between them exhibits weak sensitivity to word order and structural differences in given sentences. A single similarity score further makes the comparison process hard to interpret. Therefore, we here propose to combine sentence encoders with an alignment component by representing each sentence as a list of predicate-argument spans (where their span representations are derived from sentence encoders), and decomposing the sentence-level meaning comparison into the alignment between their spans for paraphrase identification tasks. Empirical results show that the alignment component brings in both improved performance and interpretability for various sentence encoders. After closer investigation, the proposed approach indicates increased sensitivity to structural difference and enhanced ability to distinguish non-paraphrases with high lexical overlap.

## introduction sentence meaning comparison measures the semantic similarity of two sentences. specifically, the task of paraphrase identification binarises the similarity as paraphrase or non-paraphrase depending on whether they express similar meanings @xcite . this task benefits many natural language understanding applications, like plagiarism identification @xcite and fact checking @xcite , where it is important to detect same things said in different ways. the difference in sentence structures is important for distinguishing their meanings. however, as shown in recently, various pre-trained sentence encoders have been proposed to produce high-quality sentence embeddings for downstream usages @xcite @xcite . such embeddings are compared to derive a similarity score for different meaning comparison tasks, including paraphrase identification. though widely used, sentence encoders still face challenges from different aspects in case of meaning comparison. pre-trained models are observed to capture structural information to some extent @xcite @xcite . however, as we will demonstrate in this work, their direct comparison of two sentence vectors performs poorly on paws datasets indicating weak sensitivity to structural difference, though they achieve good performance on other general paraphrase identification datasets like msrp. in addition, the single similarity score derived from the comparison of two vectors is difficult to interpret. this thus motivates us to find a better way of utilising sentence encoders for meaning comparison. elsewhere, researchers have worked on decomposing sentence-level meaning comparison into comparisons at a lower level, such as word and phrase-level, which largely increased the interpretability @xcite @xcite . alignment is the core component in these proposed systems, where sentence units at different levels are aligned through either training signals or external linguistic clues, after which a matching score is derived for sentencelevel comparison. here, we argue that, instead of comparing sentence meaning by using sentence embeddings, it would be better to combine sentence encoders with alignment components in a structureaware way to strengthen the sensitivity to structural difference and to gain interpretability. an important aspect of sentence meaning is its predicate-argument structure, which has been utilised in machine translation @xcite and paraphrase generation @xcite . given the importance of detecting structural differences in paraphrase identification tasks, we propose to represent each sentence as a list of predicate-argument spans where span representations are derived from sentence encoders, and to decompose sentence-level meaning comparison into the direct comparison between their aligned predicate-argument spans by taking advantage of the hungarian algorithm @xcite . the sentence-level score is then derived by aggregation over their aligned spans. without re-training, the proposed alignment-based sentence encoder can be used with enhanced structure-awareness and interpretability. as pre-trained sentence encoders produce contextualised representations, two phrases of different meaning might be aligned together due to their similar syntactic structure and contexts. for example: a) harris announced on twitter that he will quit. b) james announced on twitter that he will quit. unsurprisingly, the span harris announced will be aligned to the span james announced with a high similarity score given that they share exactly the same context and syntactic structure. however, it might be problematic to consider this high similarity score when we calculate the overall score given clear clashes in the meaning at sentencelevel. in this regard, we further explore how the contextualisation affects paraphrase identification by comparing aligned phrases based on their decontextualised representations. empirical results show that the inclusion of the alignment component leads to improvements on four paraphrase identification tasks and demonstrates increased ability to detect non-paraphrases with high lexical overlap, plus an enhanced sensitivity to structural difference. upon closer investigation, we find that applying de-contextualisation to aligned phrases could further help to recognise such non-paraphrases. in summary, our contributions are as follows: ## our approach instead of generating a single-vector representation for meaning comparison based on sentence encoders, we propose to represent each sentence as a list of predicate-argument spans and use sentence encoders to provide its span representations. the comparison between two sentences is then based on the alignment between their predicate-argument spans. as depicted in figure 1 , the approach can be considered as a post-processing step and consists of the following main components: sentence encoders: the input sentences are first fed into sentence encoders to produce contextualised token representations that will later be used to create context-aware phrase representations from the last hidden layer. the phrase representation will be the basic unit of our meaning comparison method. predicate argument spans (pas): for each sentence, we first apply a bert-based semantic role labelling (srl) tagger provided by allennlp @xcite to obtain both predicates and relevant arguments for each sentence. to generate predicate argument spans, we group the predicate and its arguments together and order them according to their original position in the sentence. following is an example of predicate-argument spans from a sentence: james ate some cheese whilst thinking about the play. two predicates, ate and thinking, are extracted by the tagger. as shown in figure 2 , a number of arguments with different relations are discovered for each predicate. we further group them into predicate-argument spans. for the given sentence, we will have three spans for the predicate ate: (james, ate), (ate, some, cheese), (ate, whilst, thinking, about, the, play) and two spans for the predicate thinking: (james, thinking), (thinking, about, the, play). if no predicate or associated argument is found, we take the whole sentence itself as a long span. phrase alignment: after obtaining all predicateargument spans, we derive their span representations based on the used encoder. as previous works have shown, aligning with contextual information could achieve better performance and help with disambiguation (arase and tsujii, 2020; dou and neubig, 2021). we take the mean-pooling over all tokens in the span to produce a contextualised span representation for later alignment. the tokenization strategy in bert generates subtokens, whereas in the produced spans, we have word tokens. to align them properly, we use the same tokenizer to break the original word into subtokens and represent it as a list of sub-tokens in the span if a sub-token exists. given two collections of predicate-argument span representations, @xmath0, p 2 , ..., p m } and @xmath1, q 2 , ..., q n }, we are trying to find the best alignments between them. this can be viewed as a standard assignment problem that has been extensively handled by hungarian algorithm @xcite . a similarity matrix, c, is constructed for each pair of sentences where the row has one collection of spans and the column has another. the value for each entry cell, c mn , is the cosine similarity score between the two span, p m and q n . the task of finding the best alignment is to find alignments among two collections that give the maximum score: x is a boolean matrix where x[m, n] = 1 if span m is assigned to span n. we apply the modified jonker-volgenant algorithm foot_0 (crouse, 2016) to find the best alignments that maximise the overall score. after discovering the optimal x, we obtain a collection of aligned span pairs associated with their cosine similarity scores, @xmath2, a 2 , ..., a l }. one alignment example taken from paws is given in figure 3 . ## experiments we follow the same two-step procedure in previous work for evaluation @xcite . for vanilla sentence encoders, we first generate fixed sentence embeddings, and then derive sentence-level similarity scores by calculating the cosine similarity between two embeddings. for sentence encoders combined with the alignment component, we derive sentence-level similarity scores by aggregation over cosine scores of all aligned spans where span representations are derived from sentence encoders. otherwise specifically stated, the alignment is performed between predicate-argument spans (pas). their performances under these two scenarios are evaluated and compared. we here experiment with three widely used sentence encoders, bert-base @xcite , sentence-bert (sbert) @xcite and simcse @xcite . ## analysis to better understand the improvements, we have conducted several experiments to investigate different aspects of the proposed approach. given we are mostly interested in the performance on the two paws datasets, we only experiment and report the results on the paws_qqp and paws_wiki in the following experiments. ## conclusion in this work, we propose an approach that combines sentence encoders with an alignment component by representing sentences as lists of predicateargument spans and decomposing sentence-level meaning comparison into predicate-argument span comparison. experiments with three widely used sentence encoders show that such method leads to improvements on various paraphrase identification tasks and increases the sensitivity to word order and structural differences between two sentences. the alignment matrix can further be utilised for interpretation. we then demonstrate that applying de-contextualisation to aligned phrases could help to recognise non-paraphrases of extremely high lexical overlap. our future work includes exploring other alignment algorithms and more application scenarios for alignment-based sentence encoders.