Paraphrase-based Contrastive Learning for Sentence Pair Modeling (2025)

Abstract: To improve the performance of sentence pair modeling tasks, we propose an additional pre-training method, also known as transfer fine-tuning, for pre-trained masked language models.Pre-training for masked language modeling is not necessarily designed to bring semantically similar sentences closer together in the embedding space.Our proposed method aims to improve the performance of sentence pair modeling by applying contrastive learning to pre-trained masked language models, in which sentence embeddings of paraphrase pairs are made similar to each other.While natural language inference corpora, which are standard in previous studies on contrastive learning, are not available on a large-scale for non-English languages, our method can construct a training corpus for contrastive learning from a raw corpus and a paraphrase dictionary at a low cost.Experimental results on four sentence pair modeling tasks revealed the effectiveness of our method in both English and Japanese.

## introduction sentence pair modeling @xcite , which estimates the relationship between two texts, is an important technique for various natural language processing tasks, from semantic textual similarity estimation @xcite and recognizing textual entailment @xcite to information retrieval @xcite and question answering @xcite . for sentence pair modeling tasks, surface matching such as bag-of-words and word embeddings such as word2vec @xcite have traditionally been used, followed by task-specific neural networks @xcite , and recently fine-tuning pre-trained masked language models such as bert @xcite has become the de facto standard. however, training in masked language modeling does not necessarily bring semantically similar sentences closer together in the embedding space @xcite . therefore, to maximize the effectiveness of fine-tuning for sentence pair modeling tasks, it is useful to follow the pre-training of masked language modeling with additional pre-training to estimate the semantic relationships between texts, also known as transfer fine-tuning @xcite . one such method recently been attracting attention is contrastive learning. contrastive learning for sentence embeddings, like simcse @xcite @xcite , uses annotated corpora of natural language inference (nli) to bring embeddings of entailing and entailed sentences closer together and to separate embeddings of contradictory sentence pairs. however, while nli corpora with hundreds of thousands of sentence pairs, such as stanford nli (snli) @xcite and multi-genre nli (mnli) @xcite , are available for english, there are no large-scale nli corpora for other languages, making it difficult to obtain high-quality sentence embeddings by contrastive learning for languages other than english. to improve the performance of sentence pair modeling in various languages, we propose a method of contrastive learning that does not rely on the nli corpus. our method uses a raw corpus and a paraphrase dictionary to automatically generate a large-scale training corpus for contrastive learning at a low cost. since paraphrase dictionaries are available in many languages, foot_0 this method is widely applicable. experimental results in english and japanese revealed that the proposed method could improve the performance of the masked language models in four types of sentence pair modeling tasks (product retrieval, similarity estimation, recognizing textual entailment, and paraphrase identifi- perplexity↓ paraphrase candidates 23.6 the store provides a wide range … 30.2 the store offers a broad range … 33.5 … range of of goods and accessories. 21.4 … range of goods and accessories. (1) dictionary-based paraphrase (2) filtering candidates by perplexity paraphrase dictionary p(dst|src) dst src 0.13 provides a wide offers a wide 0.13 offers a broad offers a wide 0.18 of goods and merchandise and 0.57 goods and merchandise and (3) contrastive learning paraphrased sentence irrelevant sentences input sentence input sentence the store offers a wide range of merchandise and accessories. cation). regarding the average performance of all tasks, the proposed method achieved the best performance for both english and japanese compared to existing methods that learn paraphrases but do not use contrastive learning @xcite , contrastive learning with raw corpus or nli corpus @xcite , and state-of-the-art rankcse @xcite . ## proposed method we improve the performance of sentence pair modeling with masked language models by contrastive learning that does not rely on the nli corpus. as shown in the following steps, we boost the effectiveness of fine-tuning by conducting additional training between pre-training and fine-tuning. ## experiments we evaluated the effectiveness of the proposed method for four types of sentence pair modeling tasks in both english and japanese. ## summary and future work in this study, we proposed paraphrase-based contrastive learning to improve the performance of sentence pair modeling. our method can achieve high performance from automatically generated corpora, even though it is freed from the expensive annotation of nli corpora that traditional contrastive learning relies on. experimental results reveal performance improvements in a wide range of tasks, including product retrieval, similarity estimation, recognizing textual entailment, and paraphrase identification, in both english and japanese. our future work includes both further improvement of positive instances and negative instances. especially for positive instances, paraphrase generation could be based on machine translation or large language models.