GBT : Generative Boosting Training Approach for Paraphrase Identification (2023)

Abstract: Paraphrase Identification (PI), a task of determining whether a pair of sentences express the same meaning, is widely applied in Information Retrieval and Question Answering. Data Augmentation (DA) is proven effective in tackling the PI task. However, the majority of DA methods still suffer from two limitations: inefficiency and poor quality. In this study, we propose the Generative Boosting Training (GBT) approach for PI. GBT designs a boosting learning method for a single model based on the human learning process, utilizing seq2seq model to perform DA on misclassified instances periodically. We conduct experiments on the benchmark corpora QQP and LCQMC, towards both English and Chinese PI tasks. Experimental results show that our method yields significant improvements on a variety of Pre-trained Language Model (PLM) based baselines with good efficiency and effectiveness. It is noteworthy that a single BERT model (with a linear classifier) can outperform the state-of-the-art PI models with the boosting of GBT.

## introduction pi can be boiled down to a binary classification task, aiming to determine whether pair of sentences convey the same or similar meaning. it is a fundamental natural language understanding task with non-trivial challenges, serving as a practical technique in the field of information retrieval and question answering @xcite @xcite . a variety of neural pi approaches have been proposed @xcite @xcite @xcite . recently, the large plmbased models are leveraged as crucial supportive encoders for neural pi @xcite @xcite @xcite sentences label edit distance case type 淘宝邮政编码怎么填？ (how to fill in the zip code of taobao?) 0 1 ibh0 淘宝邮政编码怎么改？ (how to modify the zip code of taobao?) 管理者怎样才能让员工服从？ (how can managers get employees to obey?) 1 8 ibh1 怎样使员工服从领导？ (how to make employees obey the leader?) no matter whether in the english or chinese scenario, the advanced pi models still suffer from two types of cases, as shown in advanced da methods such as adversarial training have been proven to be effective in solving such cases @xcite @xcite . however, advanced adversarial methods suffer from two limitations. the first one is inefficiency. the majority of these methods are based on greedy-based synonym replacement @xcite @xcite @xcite . they first need to use external knowledge bases (e.g., wordnet) to obtain lists of synonyms. and after that, a greedy-based search strategy is used to choose suitable words, which brings serious time costs. the second one is poor quality. previous work @xcite has proven most of them may result in generating unnatural and unreadable sentences, or even huge semantic shifts from the original ones. with the continuous improvement of plms, generative models, such as bart @xcite , are capable of generating sentences that are in line with human expression habits with acceptable time costs. despite the fact that using seq2seq as a da method often requires additional data or special data conditions, pi data has a naturally generative feature with source and target sentences, making seq2seq a great da method. we suggest that the existing pi models can be further improved in a simple and effective way by strengthening learning on misclassified instances. inspired by the human learning process, we believe that, just as humans learn knowledge from books, the model aims to learn previously unknown knowledge from training data. for the knowledge that is difficult to understand, we need to improve our understanding and cognition through repeated and extended learning, and so are the models. therefore, in this study, we propose the generative boosting training (gbt) method for single model, which utilizes seq2seq model as the da method serving for our boosting learning algorithm. during the training process, gbt performs da on the instances that the model did wrong in a period of training steps, and enforces the model to be retrained on these instances. in this way, gbt can enable the model to correct its errors and enhance its ability to solve hard cases in a timely manner, just like human learning. experiments on the benchmarks of english qqp and chinese lcqmc show that gbt improves var-ious plm models. it is noteworthy that a baseline model (with only a plm encoder and a linear classifier) can outperform the advanced models with the boosting of gbt. our contributions can be summarized as follow: the representation-based approaches use a siamese architecture to encode two sentences into independent vectors of high-level features. on this basis, the semantic similarity is obtained by both feature vectors. @xcite propose a siamese cnn structure, and they verify the effectiveness of cosine similarity, euclidean distance and elementwise difference. @xcite decompose two sentences to the sets of similar and different tokens to extract similar and distinguishable features by cnn. @xcite focus on the ambiguity caused by chinese word segmentation and propose a lattice-based cnn model (lcns). @xcite 's let-bert perceives multi-granularity interaction based on word lattice graph, where graph networks and external knowledge hownet @xcite are used, achieving state-of-theart performance in chinese pi. an interaction-based pi model is designed to perceive interaction features between sentences, and encode such information into the representations. specifically, @xcite propose a bimpm model under the matching-aggregation framework, which conducts one-to-many token-level matching operations to obtain interaction information. esim model proposed by @xcite performs interactive inference based on lstm representations. esim sharpens the interaction information by means of element-wise dot product and subtraction. inspired by resnet @xcite , @xcite propose a densely-connected recurrent and co-attentive network (drcn), which combines residual operation with rnn and attention mechanism. @xcite propose a relation learning network (r 2 -net) based on bert, which is characterized by interactive relationship perception of multi-granularity linguistic units. the recently proposed isg-bert @xcite integrates syntactic alignments and semantic matching signals of two sentences into an association graph to gain a fine granularity matching process. da techniques such as adversarial training have received keen attention in solving semantic matching tasks. @xcite propose a probability weighted word saliency (pwws), determining word substitution order by the word saliency and weighted by the classification probability, where wordnet @xcite is used for synonym replacement. similar to pwws, @xcite propose textbugger, which finds word substitution order by jacobian matrix and considers both character and token-level modification. the bert-attack proposed by @xcite also applies a greedy word saliency strategy to find vulnerable words. it uses bert-mlm for word generation instead of synonyms. with the advancement of auto-regressive plm, the seq2seq model is also used as a da method in various scenarios @xcite @xcite . however, seq2seq is not generalized as a da method such as synonym replacement, because it requires the data to have a generative condition, which is relatively strict. in this paper, we explore a generative augmentation method suitable for pi. ## approach let us first define the pi task in a formal way. given a pair of sentences we suggest that the learning process of the model closely parallels that of humans. the goal is to learn previously unknown knowledge from one instance, reinforce impressions, and use the learned knowledge to infer other instances. in addition, models need to timely correct mistakes during the learning process. in light of this, we devise a simple but effective training method gbt to mimic the human learning process. first, we leverage the inherent properties of pi data to transfer it into seq2seq training data, enabling the training of two independent generators. during training, both generators need to perform da on the cases where the model made mistakes. subsequently, we re-train the model on the augmented data at a certain interval. ## conclusion in this study, we propose a generative boosting training (gbt) approach for pi. gbt combines the generative da method with our boosting learning algorithm, which enables the model to correct mistakes and perform expanded learning in a timely manner during the learning process. gbt offers several advantages: high-quality and efficient da for pi, simple and effective boosting learning algorithm, and state-of-the-art matching performance. experimental results on both english and chinese benchmarks verified the effectiveness and efficiency of gbt. in the future, we will explore more generalized generative da methods to broaden the application of gbt to various tasks.