Predicate-Argument Based Bi-Encoder for Paraphrase Identification (2022)

Abstract: Paraphrase identification involves identifying whether a pair of sentences express the same or similar meanings. While cross-encoders have achieved high performances across several benchmarks, bi-encoders such as SBERT have been widely applied to sentence pair tasks. They exhibit substantially lower computation complexity and are better suited to symmetric tasks. In this work, we adopt a bi-encoder approach to the paraphrase identification task, and investigate the impact of explicitly incorporating predicate-argument information into SBERT through weighted aggregation. Experiments on six paraphrase identification datasets demonstrate that, with a minimal increase in parameters, the proposed model is able to outperform SBERT/SRoBERTa significantly. Further, ablation studies reveal that the predicate-argument based component plays a significant role in the performance gain.

## introduction paraphrases are sentences that express the same or similar meanings with different wording @xcite . paraphrase pairs are either fully or largely semantically equivalent. for example: a) marriage equality law passed in rhode island b) rhode island becomes the 10th state to enact marriage equality it is generally considered to be a symmetric task where the paraphrase relation holds in both directions @xcite . since word order and sentence structure are crucial in determining sentence meaning, effective paraphrase models must be structure-aware and word order sensitive. in light of this, paraphrase datasets have been created that are specifically designed to encourage models to consider structural differences @xcite . for example, pit2015 @xcite consists of paraphrase pairs that are lexically diverse and non-paraphrase pairs that are lexically similar but semantically dissimilar. there are generally two pre-trained based approaches for sentence pair tasks such as paraphrase identification. the first is the cross-encoder approach, which involves concatenating the two input sentences and performing full-attention over the input. the second is the bi-encoder approach, which adopts a conjoined twin network structure and maps each sentence onto separate representations, which can then be compared using similarity measures such as cosine. though typical cross-encoders like bert @xcite and roberta @xcite have set stateof-the-art performance on various sentence pair tasks @xcite , they still face challenges from both extreme computational overhead for many use cases @xcite and inconsistent predictions (ranging from 2.66% to 8.46% depending on specific datasets) when dealing with symmetric tasks @xcite . in contrast, a bi-encoder approach such as sentence-bert (sbert) (reimers and gurevych, 2019) encodes sentences separately and generates high-quality embeddings for each of them. this architecture enables sentence embeddings to be pre-computed, supporting efficient indexing and comparison between different sequences. due to the nature of bi-encoders, the symmetry property will be preserved as long as no asymmetry is introduced in subsequent layers. these properties make bi-encoders appealing for the paraphrase identification task. accordingly, here, we focus on biencoders rather than cross-encoders. one downside of sbert is that it only adopts a very simple strategy, which is mean-pooling over all tokens, to generate sentence embeddings. as previously discussed, models should ideally be sensitive to any structural differences between two sentences. relational graph convolutional networks (rgcns) @xcite have been used to introduce structural information (e.g. dependency/semantic parse trees) into sbert and improvements have been reported on unsupervised similarity comparison tasks @xcite . one drawback of rgcns is the size of the parameter space. for example, a single-layer rgcn can involve more than 30 million parameters. furthermore, as we will demonstrate, the performance gain on different paraphrase identification datasets is not consistent. an important aspect of sentence meaning concerns its predicate-argument structure. this has been utilised to generate paraphrases @xcite and to compare sentence meanings @xcite . inspired by the self-explain model @xcite which uses a span-based framework to generate sentence embeddings, we propose a method that effectively introduces sentence structure into sbert via the aggregation of predicate-argument spans. this self-attention based aggregation allows us to gain benefits with minimal increased cost in terms of additional parameters. empirical results indicate that the proposed model yields improvements on six benchmarks for paraphrase identification. upon closer investigation, we find the predicate-argument span (pas) component plays a crucial role in the performance gains and can be easily generalised to other models. ## model our proposed model adopts the same conjoined twin architecture as sbert and turns focus to the predicate-argument structure of the given sentence. as depicted in figure 1 , the model consists of different components: bert: each sentence is first fed into the pretrained bert-base model to produce both a sentence representation, by applying mean-pooling over all token representations from the last hidden layer, and an original contextualised sequencelength token representation, which is used to derive predicate-argument span representations. ## experiments we compare our model with sbert, sroberta foot_0 and the sbert-rgcn @xcite which utilises rgcn to incorporate structures into sbert with an introduction of 32 million extra parameters foot_1 . the original sentence-pair aggregation strategy of these models is @xcite . we modify this to (|u-v|, u * v) as discussed in section 3, but we retain the original notation. we adopt their structures and directly fine-tune the whole model on downstream tasks from the original bert/roberta checkpoints. we considered two strategies to apply sbert on classification inference. one involved finding the optimal similarity threshold on the development set and then applying it on the test set, while the other involved directly using the trained classifier. in this paper, we adopted the latter approach since we find it gave improved and more robust results. ## analysis in order to better understand how the performance gain is achieved, we have carried out several experiments to investigate different aspects of the proposed model. the following experiments are conducted only with sbert, since we would expect similar results with sroberta.