Assessing the Eligibility of Backtranslated Samples Based on Semantic Similarity for the Paraphrase Identification Task (2021)

Abstract: In the domain of natural language augmentation, the eligibility of generated samples remains not well understood. To gather insights around this eligibility issue, we apply a transformer-based similarity calculation within the BET framework based on backtranslation, in the context of automated paraphrase detection. While providing a rigorous statistical foundation to BET, we push their results by analyzing statistically the impacts of the level of qualification, and several sample sizes. We conducted a vast amount of experiments on the MRPC corpus using six pre-trained models: BERT, XLNet, Albert, RoBERTa, Electra, and DeBerta. We show that our method improves significantly these “base” models while using only a fraction of the corpus. Our results suggest that using some of those smaller pre-trained models, namely RoBERTa base and Electra base, helps us reach F1 scores very close to their large counterparts, as reported on the GLUE benchmark. On top of acting as a regularizer, the proposed method is efficient in dealing with data scarcity with improvements of around 3% in F1 score for most pre-trained models, and more than 7.5% in the case of Electra.

## introduction natural language processing (nlp) tasks require sufficiently large datasets to achieve the maximum robustness of the trained models. low sizes of data pose the risks of hindering the models' convergence during the training process, which leads to less accurate predictions (e.g. classification) or generations (e.g. translations). on the other hand, the provision of high-quality labelled data is often very expensive both in terms of money and time. as a result, nlp scientists seek alternative methods to tackle this issue. one of the solutions is the application of data augmentation techniques. these methods help considerably to alleviate insufficiencies regarding the quantity of labelled data and the expertise to annotate the data. these augmentation techniques are also proved to induce a regularization effect during the training of nlp models to avoid overfitting, most notably, on surface cues. in this paper, we examine the impact of adding a post-processing stage after applying such data augmentation technique to assess the eligibility of the generated samples. we intend to run our analysis in automated paraphrase identification. in this regard, we increase the size of the paraphrase data through a backtranslation method called bet @xcite . in particular, we conduct the following experiments: the remainder of this paper is structured as follows. in section 2, we describe the previous works in natural language augmentation. in section 3, we explain our methodology in terms of the dataset and our overall pipeline. next, we illustrate and discuss the results. finally, we summarize our findings and talk about the possible future research avenues. using sentence-transformers e to encode the utterances as vectors, we estimate the qualification given a threshold t by measuring the cosine similarity sim between the generated sentences and the original ones. the utterance named sentence 2 corresponds to the column name inside the mrpc dataset, which is identified as the paraphrase. ## related work data augmentation has been intensively explored in computer vision given its straightforward geometrical nature, especially image processing. according to @xcite , the popular techniques in this field are cropping, flipping, and colour jittering. from a natural language processing standpoint, many authors noted that the natural language augmentation methods (nla) either attempt to preserve the meaning and structure after the data augmentation process @xcite @xcite @xcite or to modify the tokens without taking into account the overall structure of the language @xcite . @xcite classified the techniques into the following categories: the original examples are changed (rewritten) based on a set of pre-defined rules. for instance, @xcite applied random insertion, deletion, and swap on the tokens of the sentences. to the best of our knowledge, none of the papers in the aforementioned categories has analyzed the effect of a post-processing stage so far. only @xcite and @xcite highlighted the necessity for filtering out the backtranslation outputs to assess the data augmentation validity, without conducting any specific experiment to support the claim. in the current paper, we closely set our work on the bet framework proposed by @xcite , on top of which we enrich the meaning preserving aspect with a semantic similarity stage. their original approach uses a modelbased technique by applying backtranslation on ten intermediary languages to obtain a soft data augmentation. thus, they generate ten times the amount of original data. then, they analyzed the resulting improvements on the paraphrase detection task as external validation. they tested various pre-trained models: bert @xcite , xlnet @xcite , roberta @xcite and albert @xcite . in this work, we carry out the experiments with the same pre-trained models adding the most recent ones: electra @xcite and deberta @xcite . there are still other closely related works such as @xcite , but the authors used neural network architectures such as lstms and cnns through exploiting a set of hand-crafted features on the mrpc, quora and semeval datasets. ## results and discussion as we mentioned in section 1, we evaluated our natural language augmentation approach on bert @xcite , xlnet @xcite , roberta @xcite , albert @xcite , electra @xcite and deberta @xcite . we performed our experiments iteratively beginning with the sampling of only 100 random examples from the original mrpc dataset up to reaching the whole trainset. we selected ten sample sizes to cover low-data regime situations (100 -1,000) and high-data regime situations (1,000 -3,668). we leveraged the huggingface foot_0 transformers library and the sentence-transformers library for all our fine-tuning and filtering experiments. we fixed the training configuration to well-known hyperparameters for this task based on huggingface's recommendations. we left a granular optimization of the hyperparameters for future works. thus, our experimental setup is as follows: in figure 4 , we illustrate the f-1 scores for all six models across all the sampling sizes. we display four curves: baseline (plain mrpc without bet), bet filtered 0.95, bet filtered 0.9, and bet filtered 0.8. we also added the f1 scores (dashed black lines) reported by their original authors for the corresponding large models. as a first general observation, we observe that all the baseline curves have approximately an s-shaped trend, in which sharp variations occur. in contrast, the bet filtered lines are smooth logarithmic-like growth, mostly all above the baseline curve. we further note that the higher we fix the similarity threshold, the bigger the gains we have. some models like roberta and deberta have gained between 0.04 and 0.08 in the sample size region between 500 and 1,500 samples. we directly provided in figure 6 the f-1 scores gain g in percent computed by comparing the bet filtered 0.95 to the baseline. we used the equation 3. we first note that most models are around 3% gain in the data scarcity region. for the electra base model, we observe a maximum peak of 7.6% gain in f-1 score for a 750 sample size. the second highest peak is reached by albert with nearly 5% between 500 samples and 1,500 samples. when we look at the sizes near the full dataset, we can't discern a clear portrait, despite less gain overall. we measured around 1% for roberta and electra, which lead in absolute to results near their large equivalent. two cases are slightly below the 0% -bert and albert. xlnet and deberta seem to be in between those extremums. therefore, the trend is moving from high gains in the low-data region (250 to 2,000 samples) to lower gains at higher sample sizes (3,000 to 3,668 samples). we also checked the p-values from the student's t-test between all the augmented f-1 scores and the baseline ones, in figure 5 . in statistics, we are usually advised for a minimum of about 35 runs to benefit from the law of large numbers. given the long training times of transformer models and the number of configurations set by our methodology, we limited our experiments to ten runs. however, we note it is already twice the usual five runs used in the literature with these models. the resulting comparison using the p-values is therefore limited. yet, we observe that in the case of the bet filtered 0.95 mostly all f-1 scores are strongly significant below a p-value of 0.05 (dashed black line). however, we note less significant results at 100 samples, and some at high sample sizes. those regions, as well as the bet filtered 0.8 and bet filtered 0.9, would require further runs to conclude statistically the t-test. we finally highlight that, in the case of roberta and electra in the large sample size region (from 3,000 to 3,668), the results reaching near their large counterparts are significant. to have an idea of the underlying influences behind the reported f-1 scores, we also provide the values for precision and recall in figures 7 and 8 . in figure 7 , we observe in many cases a reduction in precision. however, in the low sample sizes -below 1,500 samples -, we notice gains in precision between 0.03 and 0.15. furthermore, we report that the lower the similarity filter is(0.8 and 0.9), the more we tend to degrade the precision compared to the baseline. we remark that all the bet filtered 0.95 curves are surpassing the baseline precision. we mentioned as a first hypothesis that a higher threshold on the similarity scores would induce a higher quality of the generated samplesleading logically to a rise in precision. therefore, we confirm the validity of this hypothesis based on its impact on the precision curves. in figure 8 , we show the sensitivity curves, on which we denote two observations. first, as expected generally with data augmentation in nlp, we note an overall gain in recall when applying bet from a couple of percent up to 0.05. we observe that this gain tends to lower as the similarity threshold gets higher, but remains above the base-line. the pre-trained models that benefit the most in terms of sensitivity are respectively bert, de-berta, albert and roberta. xlnet and electra obtained very low improvements on the sensitivity metric. when looking below a 1,000 sample sizes, we notice a drastic drop in recall from the baseline to any of the bet curves. nonetheless, we rationalize that the models tend to declare a paraphrase too often. we conclude that this issue is solved by applying any backtranslation. ## conclusion and future work in this paper, we described a method based on backtranslation which is followed by a filtering stage to keep the most eligible examples. we increased the f-1 scores on the automatic paraphrase detection task by up to 7.6% compared to the baseline using only a fraction of the original dataset. furthermore, we demonstrated that this approach limits the gain in recall while avoiding degrading the precision, which results in the best f-1 scores. with the augmentation of the full dataset using roberta base and electra base, we achieved results that are close to the reported glue benchmark scores, while the original authors were using their corresponding large versions. in conclusion, pre-trained transformer models have very good transfer-learning capabilities, but they still largely benefit from the support of high-quality natural language augmentation, both to enrich very small datasets and to alleviate the overfit on surface cues. in future work, we will extend this work to the other paraphrase corpus as well as to the other nlp tasks such as multi-class classification.